Evaluating the Capability of LLMs in Identifying Compilation Errors in Configurable Systems

TL;DR

This work tackles the challenge of identifying compilation errors in configurable systems, where traditional compilers inspect a single configuration and variability-aware tooling can be costly. It evaluates three Large Language Models (ChatGPT4, Le Chat Mistral, Gemini Advanced 1.5) using a Goal-Question-Metric framework on 50 small programs and 30 configurable C systems across 17 error types. The findings show ChatGPT4 achieving the highest detection and correction rates (e.g., 41/50 detections and 44/50 corrections for products; 28/30 configurable systems with CE), while Le Chat Mistral and Gemini lag behind, especially in semantic error detection; explanations from ChatGPT4 are generally more thorough. Overall, the study demonstrates the potential of LLMs to aid developers in detecting and explaining compilation errors in configurable systems, while highlighting limitations and avenues for future work, including scaling to larger real-world systems and exploring more prompts and models.

Abstract

Compilation is an important process in developing configurable systems, such as Linux. However, identifying compilation errors in configurable systems is not straightforward because traditional compilers are not variability-aware. Previous approaches that detect some of these compilation errors often rely on advanced techniques that require significant effort from programmers. This study evaluates the efficacy of Large Language Models (LLMs), specifically ChatGPT4, Le Chat Mistral and Gemini Advanced 1.5, in identifying compilation errors in configurable systems. Initially, we evaluate 50 small products in C++, Java, and C languages, followed by 30 small configurable systems in C, covering 17 different types of compilation errors. ChatGPT4 successfully identified most compilation errors in individual products and in configurable systems, while Le Chat Mistral and Gemini Advanced 1.5 detected some of them. LLMs have shown potential in assisting developers in identifying compilation errors in configurable systems.