Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript

TL;DR

Flaky tests undermine reliability by producing non-deterministic outcomes, with test order dependency historically prominent in Java and Python but less explored in JavaScript. The authors introduce JS-TOD, a three-level randomized reordering approach (tests, describe blocks, test suites) coupled with repeated reruns to detect order-dependent tests in Jest-based projects. In an evaluation of 81 GitHub projects, they identify 55 order-dependent tests across 10 projects, driven primarily by shared files and, notably, shared mocking state—a newly highlighted mechanism in this context. The work provides actionable insights for mitigating flakiness, demonstrates the utility of test-order stress testing, and offers a replication package to promote reproducing and extending these findings.

Abstract

Flaky tests pose a significant issue for software testing. A test with a non-deterministic outcome may undermine the reliability of the testing process, making tests untrustworthy. Previous research has identified test order dependency as one of the most prevalent causes of flakiness, particularly in Java and Python. However, little is known about test order dependency in JavaScript tests. This paper aims to investigate test order dependency in JavaScript projects that use Jest, a widely used JavaScript testing framework. We implemented a systematic approach to randomise tests, test suites and describe blocks and produced 10 unique test reorders for each level. We reran each order 10 times (100 reruns for each test suite/project) and recorded any changes in test outcomes. We then manually analysed each case that showed flaky outcomes to determine the cause of flakiness. We examined our detection approach on a dataset of 81 projects obtained from GitHub. Our results revealed 55 order-dependent tests across 10 projects. Most order-dependent tests (52) occurred between tests, while the remaining three occurred between describe blocks. Those order-dependent tests are caused by either shared files (13) or shared mocking state (42) between tests. While sharing files is a known cause of order-dependent tests in other languages, our results underline a new cause (shared mocking state) that was not reported previously

Figures (2)

• Figure 1: Different levels of test order dependency with passed tests in green and failed tests in red
• Figure 2: An overview of the data collection and analysis process