Configuration Validation with Large Language Models
Xinyu Lian, Yinfang Chen, Runxiang Cheng, Jie Huang, Parth Thakkar, Minjia Zhang, Tianyin Xu
TL;DR
This paper investigates the feasibility of using large language models (LLMs) for configuration validation and introduces Ciri, an open framework that leverages prompt engineering, few-shot learning, and result voting to detect misconfigurations from configuration files or diffs. Through experiments on eight LLMs across ten open-source projects, Ciri achieves strong file-level and competitive parameter-level accuracy, notably detecting 45 of 51 real-world misconfigurations in a real dataset and outperforming prior learning-based methods in speed. The study also characterizes design trade-offs, such as the benefits of mixed-shot prompts, code augmentation, and code-specialized LLMs, while highlighting challenges with dependency/version misconfigurations and biases toward popular parameters. The results suggest that LLM-based configuration validators can provide rapid, actionable feedback to developers, complementing traditional validators and configuration testing, with an open platform for ongoing research and improvement.
Abstract
Misconfigurations are major causes of software failures. Existing practices rely on developer-written rules or test cases to validate configurations, which are expensive. Machine learning (ML) for configuration validation is considered a promising direction, but has been facing challenges such as the need of large-scale field data and system-specific models. Recent advances in Large Language Models (LLMs) show promise in addressing some of the long-lasting limitations of ML-based configuration validation. We present a first analysis on the feasibility and effectiveness of using LLMs for configuration validation. We empirically evaluate LLMs as configuration validators by developing a generic LLM-based configuration validation framework, named Ciri. Ciri employs effective prompt engineering with few-shot learning based on both valid configuration and misconfiguration data. Ciri checks outputs from LLMs when producing results, addressing hallucination and nondeterminism of LLMs. We evaluate Ciri's validation effectiveness on eight popular LLMs using configuration data of ten widely deployed open-source systems. Our analysis (1) confirms the potential of using LLMs for configuration validation, (2) explores design space of LLMbased validators like Ciri, and (3) reveals open challenges such as ineffectiveness in detecting certain types of misconfigurations and biases towards popular configuration parameters.