Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization

Zhaorui Yang, Haichao Zhu, Qian Zhang, Rajiv Gupta, Ashish Kundu

Abstract

Fault localization identifies program locations responsible for observed failures. Existing techniques rank suspicious code using syntactic spectra--signals derived from execution structure such as statement coverage, control-flow divergence, or dependency reachability. These signals collapse for semantic bugs, where failing and passing executions follow identical code paths and differ only in whether semantic intent is satisfied. Recent LLM-based approaches introduce semantic reasoning but produce stochastic, unverifiable outputs that cannot be systematically cross-referenced across tests or distinguish root causes from cascading effects. We present SemLoc, a fault localization framework based on structured semantic grounding. SemLoc converts free-form LLM reasoning into a closed intermediate representation that binds each inferred property to a typed program anchor, enabling runtime checking and attribution to program structure. It executes instrumented programs to construct a semantic violation spectrum--a constraint-by-test matrix--from which suspiciousness scores are derived analogously to coverage-based methods. A counterfactual verification step further prunes over-approximate constraints and isolates primary causal violations. We evaluate SemLoc on SemFault-250, a corpus of 250 Python programs with single semantic faults. SemLoc outperforms five coverage-, reduction-, and LLM-based baselines, achieving Top-1 accuracy of 42.8% and Top-3 of 68%, while reducing inspection to 7.6% of executable lines. Counterfactual verification provides an additional 12% accuracy gain and identifies primary causal semantic constraints.

SemLoc: Structured Grounding of Free-Form LLM Reasoning for Fault Localization

Abstract

Fault localization identifies program locations responsible for observed failures. Existing techniques rank suspicious code using syntactic spectra--signals derived from execution structure such as statement coverage, control-flow divergence, or dependency reachability. These signals collapse for semantic bugs, where failing and passing executions follow identical code paths and differ only in whether semantic intent is satisfied. Recent LLM-based approaches introduce semantic reasoning but produce stochastic, unverifiable outputs that cannot be systematically cross-referenced across tests or distinguish root causes from cascading effects. We present SemLoc, a fault localization framework based on structured semantic grounding. SemLoc converts free-form LLM reasoning into a closed intermediate representation that binds each inferred property to a typed program anchor, enabling runtime checking and attribution to program structure. It executes instrumented programs to construct a semantic violation spectrum--a constraint-by-test matrix--from which suspiciousness scores are derived analogously to coverage-based methods. A counterfactual verification step further prunes over-approximate constraints and isolates primary causal violations. We evaluate SemLoc on SemFault-250, a corpus of 250 Python programs with single semantic faults. SemLoc outperforms five coverage-, reduction-, and LLM-based baselines, achieving Top-1 accuracy of 42.8% and Top-3 of 68%, while reducing inspection to 7.6% of executable lines. Counterfactual verification provides an additional 12% accuracy gain and identifies primary causal semantic constraints.

Paper Structure

This paper contains 13 sections, 2 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivating Example
  3. Methodology
  4. Semantic Constraint Intermediate Representation
  5. Structural Anchor Extraction
  6. Agentic Constraint Inference
  7. Semantic Spectrum Analysis
  8. Counterfactual Verification
  9. Working Example
  10. Evaluation
  11. Related Work
  12. Conclusion
  13. Acknowledgement

Figures (4)

  • Figure 1: SemLoc infers semantic constraints anchored to program structures, localizes faulty lines using semantic spectrum analysis, and applies counterfactual verification to handle imprecise constraints.
  • Figure 2: Example of a Softmax Function
  • Figure 3: Condensed constraint inference prompt structure. SemLoc sends the LLM a structured prompt with four sections: task description with output schema, anchor rules grounded to SSA form, program source with SSA transformation, and test cases.
  • Figure 4: Constraint distribution per program, sorted by number of violations. Constraints are classified by violation counts: discriminative ($ef{>}0, ep{=}0$), over-approximate ($ef{>}0, ep{>}0$), and irrelevant ($ef{=}0$). Most programs contain violated constraints, with over half being discriminative.