The Same Only Different: On Information Modality for Configuration Performance Analysis
Hongyuan Liang, Yue Huang, Tao Chen
TL;DR
The paper addresses how information modality (manual versus code) influences configuration performance analysis. It conducts a large-scale, two-phase empirical study across 10 widely used systems to compare manual and code in identifying performance-sensitive options and extracting dependencies, and to evaluate existing automated tools. Key findings show that fusing manual and code information yields the best performance, that code generally better detects dependencies while manuals reveal edge cases, and that current tools using single modalities struggle to match human analysis. The work highlights practical implications for multi-modal analysis design, dataset release, and future interactive tools to support configuration tuning and management in complex software systems.
Abstract
Configuration in software systems helps to ensure efficient operation and meet diverse user needs. Yet, some, if not all, configuration options have profound implications for the system's performance. Configuration performance analysis, wherein the key is to understand (or infer) the configuration options' relations and their impacts on performance, is crucial. Two major modalities exist that serve as the source information in the analysis: either the manual or source code. However, it remains unclear what roles they play in configuration performance analysis. Much work that relies on manuals claims their benefits of information richness and naturalness; while work that trusts the source code more prefers the structural information provided therein and criticizes the timeliness of manuals. To fill such a gap, in this paper, we conduct an extensive empirical study over 10 systems, covering 1,694 options, 106,798 words in the manual, and 22,859,552 lines-of-code for investigating the usefulness of manual and code in two important tasks of configuration performance analysis, namely performance-sensitive options identification and the associated dependencies extraction. We reveal several new findings and insights, such as it is beneficial to fuse the manual and code modalities for both tasks; the current automated tools that rely on a single modality are far from being practically useful and generally remain incomparable to human analysis. All those pave the way for further advancing configuration performance analysis.