Graph Neural Networks for Vulnerability Detection: A Counterfactual Explanation

TL;DR

This work addresses explainability in GNN-based vulnerability detection by introducing CFExplainer, a counterfactual reasoning-based explainer that identifies minimal perturbations to code graphs to flip predictions. By learning a differentiable edge mask, CFExplainer generates minimal, actionable counterfactual explanations that reveal root causes of detected vulnerabilities, going beyond traditional factual explanations. Extensive experiments on Big-Vul and four GNNs show that CFExplainer outperforms state-of-the-art factual explainers on vulnerability-oriented and model-oriented metrics and demonstrates robustness through a case study and parameter analysis. The approach enables developers to understand and fix vulnerabilities more effectively, with potential applicability to broader code-analysis tasks such as bug detection and code localization.

Abstract

Vulnerability detection is crucial for ensuring the security and reliability of software systems. Recently, Graph Neural Networks (GNNs) have emerged as a prominent code embedding approach for vulnerability detection, owing to their ability to capture the underlying semantic structure of source code. However, GNNs face significant challenges in explainability due to their inherently black-box nature. To this end, several factual reasoning-based explainers have been proposed. These explainers provide explanations for the predictions made by GNNs by analyzing the key features that contribute to the outcomes. We argue that these factual reasoning-based explanations cannot answer critical what-if questions: What would happen to the GNN's decision if we were to alter the code graph into alternative structures? Inspired by advancements of counterfactual reasoning in artificial intelligence, we propose CFExplainer, a novel counterfactual explainer for GNN-based vulnerability detection. Unlike factual reasoning-based explainers, CFExplainer seeks the minimal perturbation to the input code graph that leads to a change in the prediction, thereby addressing the what-if questions for vulnerability detection. We term this perturbation a counterfactual explanation, which can pinpoint the root causes of the detected vulnerability and furnish valuable insights for developers to undertake appropriate actions for fixing the vulnerability. Extensive experiments on four GNN-based vulnerability detection models demonstrate the effectiveness of CFExplainer over existing state-of-the-art factual reasoning-based explainers.

Figures (5)

• Figure 1: Illustration of factual reasoning-based explanation (right middle) and what-if analysis (right bottom).
• Figure 2: An overview of our proposed counterfactual reasoning framework.
• Figure 3: Comparison for the model-oriented evaluation results of explainers.
• Figure 4: A parameter analysis on the hyper-parameter $\alpha$.
• Figure 5: A case study on the CVE-2017-13001 vulnerability in the tcpdump project.