Explainable Fault Localization for Programming Assignments via LLM-Guided Annotation

TL;DR

FLAME tackles the challenge of explainable, fine-grained fault localization in programming assignments by leveraging LLM-guided annotation and a weighted multi-model voting scheme. It collects assignment-specific context (problem statements, test outcomes, reference programs), prompts LLMs to annotate faulty lines with explanations instead of predicting line numbers, and aggregates multiple models to produce robust suspicious-line rankings. Evaluations on PADefects and Defects4J show FLAME outperforms state-of-the-art baselines and improves automated program repair, highlighting educational utility and generalization to real-world software. The approach demonstrates strong educational value through correct, informative explanations and actionable feedback while maintaining scalability to project-level submissions and larger codebases.

Abstract

Providing timely and personalized guidance for students' programming assignments, offers significant practical value for helping students complete assignments and enhance their learning. In recent years, various automated Fault Localization (FL) techniques have demonstrated promising results in identifying errors in programs. However, existing FL techniques face challenges when applied to educational contexts. Most approaches operate at the method level without explanatory feedback, resulting in granularity too coarse for students who need actionable insights to identify and fix their errors. While some approaches attempt line-level fault localization, they often depend on predicting line numbers directly in numerical form, which is ill-suited to LLMs. To address these challenges, we propose FLAME, a fine-grained, explainable Fault Localization method tailored for programming assignments via LLM-guided Annotation and Model Ensemble. FLAME leverages rich contextual information specific to programming assignments to guide LLMs in identifying faulty code lines. Instead of directly predicting line numbers, we prompt the LLM to annotate faulty code lines with detailed explanations, enhancing both localization accuracy and educational value. To further improve reliability, we introduce a weighted multi-model voting strategy that aggregates results from multiple LLMs to determine the suspiciousness of each code line. Extensive experimental results demonstrate that FLAME outperforms state-of-the-art fault localization baselines on programming assignments, successfully localizing 207 more faults at top-1 over the best-performing baseline. Beyond educational contexts, FLAME also generalizes effectively to general-purpose software codebases, outperforming all baselines on the Defects4J benchmark.

Figures (10)

• Figure 1: Overview of FLAME, including ❶ Assignment-Specific Information Collection, ❷ Fault Location Annotation, and ❸ Weighted Multi-Model Voting.
• Figure 2: An example of a problem statement, consisting of the problem description, input/output specifications, and input/output samples.
• Figure 3: An example of an annotated program. Line 7 is marked as faulty by appending a marker // @@ at the end of the line, followed by an explanation stating that i * i < n should be corrected to i * i <= n.
• Figure 4: Distribution of localized faults on PADefectsDS.
• Figure 5: Distribution of localized faults on PADefectsCT.