Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can We Classify Flaky Tests Using Only Test Code? An LLM-Based Empirical Study

Alexander Berndt, Vekil Bekmyradov, Rainer Gemulla, Marcus Kessel, Thomas Bach, Sebastian Baltes

TL;DR

This work interrogates whether large language models can classify flaky tests using only the test code without task-specific fine-tuning. Across two benchmarks (IDoFT and FlakeBench) and three prompting strategies, the best non-fine-tuned LLM performance yields weak correlations (MCC around 0.17–0.27) and notable output non-determinism, suggesting test-code alone is insufficient for reliable flakiness detection. Human judgment on 50 samples corroborates that many flakiness types require broader program context beyond the test code. The findings challenge prior claims of high performance from fine-tuned models and motivate future work using retrieval-augmented generation or agentic AI to incorporate additional context. Practically, the results imply that deploying LLM-based flakiness detectors in real-world pipelines will require richer information sources beyond test code to achieve trustworthy performance.

Abstract

Flaky tests yield inconsistent results when they are repeatedly executed on the same code revision. They interfere with automated quality assurance of code changes and hinder efficient software testing. Previous work evaluated approaches to train machine learning models to classify flaky tests based on identifiers in the test code. However, the resulting classifiers have been shown to lack generalizability, hindering their applicability in practical environments. Recently, pre-trained Large Language Models (LLMs) have shown the capability to generalize across various tasks. Thus, they represent a promising approach to address the generalizability problem of previous approaches. In this study, we evaluated three LLMs (two general-purpose models, one code-specific model) using three prompting techniques on two benchmark datasets from prior studies on flaky test classification. Furthermore, we manually investigated 50 samples from the given datasets to determine whether classifying flaky tests based only on test code is feasible for humans. Our findings indicate that LLMs struggle to classify flaky tests given only the test code. The results of our best prompt-model combination were only marginally better than random guessing. In our manual analysis, we found that the test code does not necessarily contain sufficient information for a flakiness classification. Our findings motivate future work to evaluate LLMs for flakiness classification with additional context, for example, using retrieval-augmented generation or agentic AI.

Can We Classify Flaky Tests Using Only Test Code? An LLM-Based Empirical Study

TL;DR

This work interrogates whether large language models can classify flaky tests using only the test code without task-specific fine-tuning. Across two benchmarks (IDoFT and FlakeBench) and three prompting strategies, the best non-fine-tuned LLM performance yields weak correlations (MCC around 0.17–0.27) and notable output non-determinism, suggesting test-code alone is insufficient for reliable flakiness detection. Human judgment on 50 samples corroborates that many flakiness types require broader program context beyond the test code. The findings challenge prior claims of high performance from fine-tuned models and motivate future work using retrieval-augmented generation or agentic AI to incorporate additional context. Practically, the results imply that deploying LLM-based flakiness detectors in real-world pipelines will require richer information sources beyond test code to achieve trustworthy performance.

Abstract

Flaky tests yield inconsistent results when they are repeatedly executed on the same code revision. They interfere with automated quality assurance of code changes and hinder efficient software testing. Previous work evaluated approaches to train machine learning models to classify flaky tests based on identifiers in the test code. However, the resulting classifiers have been shown to lack generalizability, hindering their applicability in practical environments. Recently, pre-trained Large Language Models (LLMs) have shown the capability to generalize across various tasks. Thus, they represent a promising approach to address the generalizability problem of previous approaches. In this study, we evaluated three LLMs (two general-purpose models, one code-specific model) using three prompting techniques on two benchmark datasets from prior studies on flaky test classification. Furthermore, we manually investigated 50 samples from the given datasets to determine whether classifying flaky tests based only on test code is feasible for humans. Our findings indicate that LLMs struggle to classify flaky tests given only the test code. The results of our best prompt-model combination were only marginally better than random guessing. In our manual analysis, we found that the test code does not necessarily contain sufficient information for a flakiness classification. Our findings motivate future work to evaluate LLMs for flakiness classification with additional context, for example, using retrieval-augmented generation or agentic AI.
Paper Structure (18 sections, 1 equation, 3 figures, 7 tables)

This paper contains 18 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. Related Work
  4. Issues of Prior LLM-based Flakiness Classifiers
  5. Methodology
  6. RQ1: Classification
  7. RQ2: Non-Determinism
  8. RQ3: Human Judgement
  9. Results
  10. RQ1: Classification
  11. RQ2: Reliability
  12. RQ3: Manual Investigation
  13. Discussion
  14. Threats to Validity
  15. Internal Validity
  16. ...and 3 more sections

Figures (3)

  • Figure 1: The structure of our $x_{CoT}$ prompt template.
  • Figure 2: Examples of reset_variable and change_assertion (diff indicates fixed version of the corresponding test).
  • Figure 3: Heatmap showing the results of the survey from two reviewers. Answers to the question "How likely would a developer proficient in Java classify this test as flaky?" range from 1 (Very likely) to 5 (Very unlikely).