Rethinking Software Misconfigurations in the Real World: An Empirical Study and Literature Analysis

TL;DR

This work tackles the gap between real-world software misconfigurations and literature by presenting an empirical study of 823 misconfigurations and a literature review spanning 2003–2023. It introduces a four-type root-cause taxonomy (constraint violation, resource unavailability, component-dependency error, misunderstanding of configuration effects) and documents detailed subtypes, supported by case examples. The authors also assess the landscape of misconfiguration troubleshooting literature, tools, and benchmarks, revealing limited public tooling and datasets and highlighting trends toward non-crash symptoms and cloud-era targets. They provide a public 823-case dataset and Docker-wrapped reproductions to aid replication, and offer practical suggestions for users and developers to close the practice–research gap. Overall, the work emphasizes the need for better benchmarks, reusable tools, and clearer correspondence between configuration processes and real-world outcomes.

Abstract

Software misconfiguration has consistently been a major reason for software failures. Over the past twenty decades, much work has been done to detect and diagnose software misconfigurations. However, there is still a gap between real-world misconfigurations and the literature. It is desirable to investigate whether existing taxonomy and tools are applicable for real-world misconfigurations in modern software. In this paper, we conduct an empirical study on 823 real-world misconfiguration issues, based on which we propose a novel classification of the root causes of software misconfigurations, i.e., constraint violation, resource unavailability, component-dependency error, and misunderstanding of configuration effects. Then, we systematically review the literature on misconfiguration troubleshooting, and study the trends of research and the practicality of the tools and datasets in this field. We find that the research targets have changed from fundamental software to advanced applications (e.g., cloud service). In the meanwhile, the research on non-crash misconfigurations such as performance degradation and security risks also has a significant growth. Despite the progress, a majority of studies lack reproducibility due to the unavailable tools and evaluation datasets. In total, only six tools and two datasets are publicly available. However, the adaptability of these tools limit their practical use on real-world misconfigurations. We also summarize the important challenges and several suggestions to facilitate the research on software misconfiguration.

Figures (6)

• Figure 1: Overview of software configuration life-cycle.
• Figure 2: Overview of our methodology.
• Figure 3: Root causes of configuration errors.
• Figure 4: Examples of misconfigurations caused by different root causes.
• Figure 5: Trends of research targets and misconfiguration symptoms.