Using a Sledgehammer to Crack a Nut? Revisiting Automated Compiler Fault Isolation
Yibiao Yang, Qingyang Li, Maolin Sun, Jiangchang Wu, Yuming Zhou
TL;DR
This paper investigates compiler fault localization, comparing traditional spectrum-based fault localization (SBFL) techniques with Basic, a practitioner-aligned, commit-history–driven baseline that uses binary search to identify the bug-inducing commit (BIC) and flags all files touched by that commit as faulty. Across 60 GCC and 60 LLVM real-world bugs, Basic often matches or exceeds SBFL performance, particularly on the critical Top-1 and Top-5 metrics, challenging the assumed superiority of SBFL methods in this domain. The study shows Basic is efficient enough for practical use and frequently complementary to SBFL methods, suggesting hybrid approaches could yield even better localization. The results advocate adopting Basic as a baseline in compiler fault localization research and emphasize the value of aligning techniques with real-world developer workflows.
Abstract
Background: Compilers are fundamental to software development, translating high-level source code into executable software systems. Faults in compilers can have severe consequences and thus effective localization and resolution of compiler bugs are crucial. Problem: In practice, developers often examine version history to identify and investigate bug-inducing commit (BIC) for fixing bugs. However, while numerous sophisticated Spectrum-Based Fault Localization (SBFL) techniques have been proposed for compiler fault isolation, their effectiveness has not been evaluated against the BIC-based strategies widely adopted in practice. Objective: This study aims to bridge this gap by directly comparing a BIC-based strategy, Basic, with representative SBFL techniques in the context of compiler fault localization. The BIC-based strategy closely aligns with common developer practices, as it directly identifies the BIC and treats the files modified in that commit as faulty candidates. Method: The Basic identifies the most recent good release and earliest bad release, and then employs a binary search to pinpoint the bug-inducing commit. All files modified in the identified commit are flagged as potentially faulty. We rigorously compare Basic against SBFL-based techniques using a benchmark consisting of 60 GCC bugs and 60 LLVM bugs. Result: Our analysis reveals that Basic performs comparably to, and in many cases outperforms, state-of-the-art SBFL-based techniques, particularly on the critical Top-1 and Top-5 ranking metrics. Conclusion: This study provides new insights into the practical effectiveness of SBFL-based techniques in real-world compiler debugging scenarios. We recommend that future research adopt Basic as a baseline when developing and evaluating new compiler fault isolation methods.