Can You Mimic Me? Exploring the Use of Android Record & Replay Tools in Debugging

TL;DR

This work provides the first comprehensive empirical evaluation of Android record-and-replay (R&R) tools for bug reproduction, across common user scenarios, non-crashing failures, crashes from bug reports, and crashes detected by automated input generation (AIG). It combines four R&R tools with large-scale datasets (Themis, AndroR2) to measure recording and replay success, revealing substantial unreliability: 17% for scenarios, 38% for failures, and 44% for crashes are not reliably replayed. The study identifies root causes—short action intervals, API incompatibilities, lazy logging, and tool limitations—and demonstrates the challenges of integrating AIG with R&R for crash reproduction, largely due to Android SDK conflicts and visualization-based replay constraints. The findings highlight critical directions for improving action recording fidelity, API coverage, and cross-tool integration, and the authors provision an open dataset to spur further research in reproducible debugging with R&R and AIG. The work has practical significance for developers seeking reliable debugging workflows and for researchers advancing robust, cross-device testing pipelines in real-world Android ecosystems.

Abstract

Android User Interface (UI) testing is a critical research area due to the ubiquity of apps and the challenges faced by developers. Record and replay (R&R) tools facilitate manual and automated UI testing by recording UI actions to execute test scenarios and replay bugs. These tools typically support (i) regression testing, (ii) non-crashing functional bug reproduction, and (iii) crashing bug reproduction. However, prior work only examines these tools in fragmented settings, lacking a comprehensive evaluation across common use cases. We address this gap by conducting an empirical study on using R&R tools to record and replay non-crashing failures, crashing bugs, and feature-based user scenarios, and explore combining R&R with automated input generation (AIG) tools to replay crashing bugs. Our study involves one industrial and three academic R&R tools, 34 scenarios from 17 apps, 90 non-crashing failures from 42 apps, and 31 crashing bugs from 17 apps. Results show that 17% of scenarios, 38% of non-crashing bugs, and 44% of crashing bugs cannot be reliably recorded and replayed, mainly due to action interval resolution, API incompatibility, and Android tooling limitations. Our findings highlight key future research directions to enhance the practical application of R&R tools.

Figures (2)

• Figure 1: An overview of the relationship of issue symptoms and root causes
• Figure 2: Number of unique crashes detected by the Themis dataset and Steps 1 and 2 of our AIG Tool experiments.