Empirical evaluation of LLMs in predicting fixes of Configuration bugs in Smart Home System

TL;DR

This study tackles the problem of repairing configuration bugs in smart-home systems by evaluating three Large Language Models (GPT-4, GPT-4o, and Claude 3.5 Sonnet) across four prompt designs. Using a HAC-derived dataset of 129 debugging issues (with 21 analyzed in-depth) and eight representative resolution strategies, the authors map model outputs to fixes and compare performance in predicting both fix strategies and concrete fixes. Key findings show GPT-4 and Claude 3.5 Sonnet achieve about 80% accuracy in strategy prediction when given descriptions plus the original script (Prompt 2), while GPT-4 demonstrates the most consistent performance across prompts; GPT-4o offers speed and cost benefits with slightly lower accuracy. The results underscore the importance of prompt design in eliciting high-quality solutions and demonstrate the potential for AI-assisted automated bug fixing in smart-home configurations, paving the way for integrated debugging tools in Home Assistant and similar platforms.

Abstract

This empirical study evaluates the effectiveness of Large Language Models (LLMs) in predicting fixes for configuration bugs in smart home systems. The research analyzes three prominent LLMs - GPT-4, GPT-4o (GPT-4 Turbo), and Claude 3.5 Sonnet - using four distinct prompt designs to assess their ability to identify appropriate fix strategies and generate correct solutions. The study utilized a dataset of 129 debugging issues from the Home Assistant Community, focusing on 21 randomly selected cases for in-depth analysis. Results demonstrate that GPT-4 and Claude 3.5 Sonnet achieved 80\% accuracy in strategy prediction when provided with both bug descriptions and original scripts. GPT-4 exhibited consistent performance across different prompt types, while GPT-4o showed advantages in speed and cost-effectiveness despite slightly lower accuracy. The findings reveal that prompt design significantly impacts model performance, with comprehensive prompts containing both description and original script yielding the best results. This research provides valuable insights for improving automated bug fixing in smart home system configurations and demonstrates the potential of LLMs in addressing configuration-related challenges.

Figures (5)

• Figure 1: Overview of the debugging dataset.
• Figure 2: Representative Resolution Strategies.
• Figure 3: Four prompt designs for generating fix strategies
• Figure 4: Four prompt designs for generating fix strategies
• Figure 5: Example suggested fix strategies by three LLMs.