WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

TL;DR

WEFix tackles UI-based flakiness in web e2e tests by automatically generating explicit waits from observed DOM mutations. It combines Mutation Recorder (test-code instrumentation and browser mutation capture) with an Oracle Generator (pruning mutations, a mutation FSM, and wait oracle construction) to insert waits that fire only when necessary. In evaluation across seven real-world projects and 122 reproduced flaky tests, WEFix fixed 98.4% of cases and reduced runtime overhead from implicit-wait baselines to about 1.25x on average, outperforming fixed implicit-wait strategies. The approach is available as an open-source npm tool, enabling developers to automatically stabilize flaky tests with minimal performance impact.

Abstract

Web end-to-end (e2e) testing evaluates the workflow of a web application. It simulates real-world user scenarios to ensure the application flows behave as expected. However, web e2e tests are notorious for being flaky, i.e., the tests can produce inconsistent results despite no changes to the code. One common type of flakiness is caused by nondeterministic execution orders between the test code and the client-side code under test. In particular, UI-based flakiness emerges as a notably prevalent and challenging issue to fix because the test code has limited knowledge about the client-side code execution. In this paper, we propose WEFix, a technique that can automatically generate fix code for UI-based flakiness in web e2e testing. The core of our approach is to leverage browser UI changes to predict the client-side code execution and generate proper wait oracles. We evaluate the effectiveness and efficiency of WEFix against 122 web e2e flaky tests from seven popular real-world projects. Our results show that WEFix dramatically reduces the overhead (from 3.7$\times$ to 1.25$\times$) while achieving a high correctness (98%).

Figures (6)

• Figure 1: The web page of the example shown in Listing \ref{['lst:example']}.
• Figure 2: Interleaved mutations triggered by two consecutive commands.
• Figure 3: The workflow of WEFix.
• Figure 4: Mutation Recorder runtime.
• Figure 5: Finite state machine (FSM) representing DOM tree status transitions via mutations $m_1,m_2,...,m_j$.