Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward Explaining Large Language Models in Software Engineering Tasks

Antonio Vitale, Khai-Nguyen Nguyen, Denys Poshyvanyk, Rocco Oliveto, Simone Scalabrino, Antonio Mastropaolo

TL;DR

This paper presents FeatureSHAP, the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. By using Shapley-based attributions over semantically meaningful input features, it yields explanations that align with how developers reason about code and documentation, demonstrated on code generation and code summarization. Through noise-injection experiments and a practitioner survey (n=37), FeatureSHAP shows strong fidelity to model behavior, reduced attribution to non-influential inputs, and meaningful usefulness for decision-making. The work lays groundwork for practical explainable AI in SE and outlines future directions for broader tasks, causal validation, and integration into development workflows.

Abstract

Recent progress in Large Language Models (LLMs) has substantially advanced the automation of software engineering (SE) tasks, enabling complex activities such as code generation and code summarization. However, the black-box nature of LLMs remains a major barrier to their adoption in high-stakes and safety-critical domains, where explainability and transparency are vital for trust, accountability, and effective human supervision. Despite increasing interest in explainable AI for software engineering, existing methods lack domain-specific explanations aligned with how practitioners reason about SE artifacts. To address this gap, we introduce FeatureSHAP, the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. Based on Shapley values, FeatureSHAP attributes model outputs to high-level input features through systematic input perturbation and task-specific similarity comparisons, while remaining compatible with both open-source and proprietary LLMs. We evaluate FeatureSHAP on two bi-modal SE tasks: code generation and code summarization. The results show that FeatureSHAP assigns less importance to irrelevant input features and produces explanations with higher fidelity than baseline methods. A practitioner survey involving 37 participants shows that FeatureSHAP helps practitioners better interpret model outputs and make more informed decisions. Collectively, FeatureSHAP represents a meaningful step toward practical explainable AI in software engineering. FeatureSHAP is available at https://github.com/deviserlab/FeatureSHAP.

Toward Explaining Large Language Models in Software Engineering Tasks

TL;DR

This paper presents FeatureSHAP, the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. By using Shapley-based attributions over semantically meaningful input features, it yields explanations that align with how developers reason about code and documentation, demonstrated on code generation and code summarization. Through noise-injection experiments and a practitioner survey (n=37), FeatureSHAP shows strong fidelity to model behavior, reduced attribution to non-influential inputs, and meaningful usefulness for decision-making. The work lays groundwork for practical explainable AI in SE and outlines future directions for broader tasks, causal validation, and integration into development workflows.

Abstract

Recent progress in Large Language Models (LLMs) has substantially advanced the automation of software engineering (SE) tasks, enabling complex activities such as code generation and code summarization. However, the black-box nature of LLMs remains a major barrier to their adoption in high-stakes and safety-critical domains, where explainability and transparency are vital for trust, accountability, and effective human supervision. Despite increasing interest in explainable AI for software engineering, existing methods lack domain-specific explanations aligned with how practitioners reason about SE artifacts. To address this gap, we introduce FeatureSHAP, the first fully automated, model-agnostic explainability framework tailored to software engineering tasks. Based on Shapley values, FeatureSHAP attributes model outputs to high-level input features through systematic input perturbation and task-specific similarity comparisons, while remaining compatible with both open-source and proprietary LLMs. We evaluate FeatureSHAP on two bi-modal SE tasks: code generation and code summarization. The results show that FeatureSHAP assigns less importance to irrelevant input features and produces explanations with higher fidelity than baseline methods. A practitioner survey involving 37 participants shows that FeatureSHAP helps practitioners better interpret model outputs and make more informed decisions. Collectively, FeatureSHAP represents a meaningful step toward practical explainable AI in software engineering. FeatureSHAP is available at https://github.com/deviserlab/FeatureSHAP.
Paper Structure (21 sections, 10 figures, 2 tables)

This paper contains 21 sections, 10 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background & Related Work
  3. Explainability and Interpretability
  4. Explainability and Interpretability in Software Engineering
  5. FeatureSHAP
  6. Study Design
  7. Context Selection
  8. Experimental Procedure for RQ1
  9. Experimental Procedure for RQ2
  10. Experimental Procedure for RQ3
  11. Results
  12. RQ1: Random Noise Attribution Scores
  13. RQ2: Cross-sample Noise Attribution Scores
  14. RQ3: FeatureSHAP Alignment with Human Perception.
  15. Implications of Our Findings and Future Work
  16. ...and 6 more sections

Figures (10)

  • Figure 1: Token‐level (top) and feature‐level (bottom) attributions for the same input; darker rectangles represent higher attribution scores.
  • Figure 2: FeatureSHAP pipeline.
  • Figure 3: Prompt for the LLM-based splitter (code generation). Here, {docstring} refers to the docstring-formatted prompt used by BigCodeBench for code generation.
  • Figure 4: Demonstration of an ICL example used in LLM-based splitter.
  • Figure 5: FeatureSHAP feature attribution for code generation. The darker the highlight, the higher the influence of that feature on the generated code.
  • ...and 5 more figures