Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flaky Tests in a Large Industrial Database Management System: An Empirical Study of Fixed Issue Reports for SAP HANA

Alexander Berndt, Thomas Bach, Sebastian Baltes

TL;DR

This study addresses flaky tests in a large industrial DBMS (SAP HANA) by using LLMs-as-annotators to label issue reports with root-cause categories. It finds concurrency to be the dominant root cause (23%) and reveals test-type differences, with system tests showing more timeouts and unit tests more platform- and isolation-related flakiness. The approach achieves substantial model-model (κ=0.78) and model-human (κ=0.63) agreement, supporting scalable, large-scale analysis, and uncovers temporal trends such as rising Async Wait and a post-change drop in timeouts. The work highlights the value of multi-label root-cause analysis for understanding complex flakiness and informs future mitigation strategies in industrial contexts.

Abstract

Flaky tests yield different results when executed multiple times for the same version of the source code. Thus, they provide an ambiguous signal about the quality of the code and interfere with the automated assessment of code changes. While a variety of factors can cause test flakiness, approaches to fix flaky tests are typically tailored to address specific causes. However, the prevalent root causes of flaky tests can vary depending on the programming language, application domain, or size of the software project. Since manually labeling flaky tests is time-consuming and tedious, this work proposes an LLMs-as-annotators approach that leverages intra- and inter-model consistency to label issue reports related to fixed flakiness issues with the relevant root cause category. This allows us to gain an overview of prevalent flakiness categories in the issue reports. We evaluated our labeling approach in the context of SAP HANA, a large industrial database management system. Our results suggest that SAP HANA's tests most commonly suffer from issues related to concurrency (23%, 130 of 559 analyzed issue reports). Moreover, our results suggest that different test types face different flakiness challenges. Therefore, we encourage future research on flakiness mitigation to consider evaluating the generalizability of proposed approaches across different test types.

Flaky Tests in a Large Industrial Database Management System: An Empirical Study of Fixed Issue Reports for SAP HANA

TL;DR

This study addresses flaky tests in a large industrial DBMS (SAP HANA) by using LLMs-as-annotators to label issue reports with root-cause categories. It finds concurrency to be the dominant root cause (23%) and reveals test-type differences, with system tests showing more timeouts and unit tests more platform- and isolation-related flakiness. The approach achieves substantial model-model (κ=0.78) and model-human (κ=0.63) agreement, supporting scalable, large-scale analysis, and uncovers temporal trends such as rising Async Wait and a post-change drop in timeouts. The work highlights the value of multi-label root-cause analysis for understanding complex flakiness and informs future mitigation strategies in industrial contexts.

Abstract

Flaky tests yield different results when executed multiple times for the same version of the source code. Thus, they provide an ambiguous signal about the quality of the code and interfere with the automated assessment of code changes. While a variety of factors can cause test flakiness, approaches to fix flaky tests are typically tailored to address specific causes. However, the prevalent root causes of flaky tests can vary depending on the programming language, application domain, or size of the software project. Since manually labeling flaky tests is time-consuming and tedious, this work proposes an LLMs-as-annotators approach that leverages intra- and inter-model consistency to label issue reports related to fixed flakiness issues with the relevant root cause category. This allows us to gain an overview of prevalent flakiness categories in the issue reports. We evaluated our labeling approach in the context of SAP HANA, a large industrial database management system. Our results suggest that SAP HANA's tests most commonly suffer from issues related to concurrency (23%, 130 of 559 analyzed issue reports). Moreover, our results suggest that different test types face different flakiness challenges. Therefore, we encourage future research on flakiness mitigation to consider evaluating the generalizability of proposed approaches across different test types.
Paper Structure (16 sections, 5 figures, 2 tables)

This paper contains 16 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Study Context
  4. Dataset
  5. Research Questions and Methodology
  6. RQ1: Root Causes
  7. Motivation
  8. Approach
  9. RQ2: Trends and Patterns
  10. Motivation
  11. Approach
  12. Results
  13. RQ1: Root Causes
  14. RQ2: Trends and Patterns
  15. Discussion
  16. ...and 1 more sections

Figures (5)

  • Figure 1: An example issue report.
  • Figure 2: Histogram showing the number of linked tests per issue report. Note the logarithmic scale of the y-axis.
  • Figure 3: Our bug report labeling approach.
  • Figure 4: Barplot showing the number of issue reports per category per test type. Includes only issue reports with more than a single linked test (n=127).
  • Figure 5: Heatmap showing the number of issue reports per category per quarter. Values are min-max-normalized within each category; absolute values are shown in parentheses. Note that some categories (rows) show high numbers throughout the period while others show more concentrated failure numbers.