EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks

TL;DR

A novel framework is designed: Evaluation of Explainable AI (EvaluateXAI), along with granular-level evaluation metrics, to automatically assess the effectiveness of rule-based XAI techniques in generating reliable and consistent explanations for ML models in software analytics tasks.

Abstract

The advancement of machine learning (ML) models has led to the development of ML-based approaches to improve numerous software engineering tasks in software maintenance and evolution. Nevertheless, research indicates that despite their potential successes, ML models may not be employed in real-world scenarios because they often remain a black box to practitioners, lacking explainability in their reasoning. Recently, various rule-based model-agnostic Explainable AI (XAI) techniques, such as PyExplainer and LIME, have been employed to explain the predictions of ML models in software analytics tasks. This paper assesses the ability of these techniques (e.g., PyExplainer and LIME) to generate reliable and consistent explanations for ML models across various software analytics tasks, including Just-in-Time (JIT) defect prediction, clone detection, and the classification of useful code review comments. Our manual investigations find inconsistencies and anomalies in the explanations generated by these techniques. Therefore, we design a novel framework: Evaluation of Explainable AI (EvaluateXAI), along with granular-level evaluation metrics, to automatically assess the effectiveness of rule-based XAI techniques in generating reliable and consistent explanations for ML models in software analytics tasks. After conducting in-depth experiments involving seven state-of-the-art ML models trained on five datasets and six evaluation metrics, we find that none of the evaluation metrics reached 100\%, indicating the unreliability of the explanations generated by XAI techniques. Additionally, PyExplainer and LIME failed to provide consistent explanations for 86.11% and 77.78% of the experimental combinations, respectively. Therefore, our experimental findings emphasize the necessity for further research in XAI to produce reliable and consistent explanations for ML models in software analytics tasks.

Figures (18)

• Figure 1: Explanations generated by PyExplainer for a single instance reveal that the model predicts the instance as a defect-introducing commit with a 77% probability, as indicated by the Risk Score.
• Figure 2: Figures \ref{['lime_a']} and \ref{['lime_b']} demonstrate that LIME produces different explanations when regenerated for the same instance. These two figures collectively visualize why the given instance is predicted as a buggy commit. Figure \ref{['lime_c']} visually explains an instance predicted as a clean commit.
• Figure 3: Explanations generated by PyExplainer for a single instance reveal that the model predicts the instance as a defect-introducing commit with a 77% probability (as indicated by the Risk Score) as shown in Figures \ref{['interpret_a']} and \ref{['interpret_b']}. On the other hand, Figures \ref{['interpret_c']}, \ref{['interpret_d']}, and \ref{['interpret_e']} depict the generated explanations when the ML model predicts the given instance as a clean commit.
• Figure 4: While changing the feature values in the guided directions, PyExplainer fails to function correctly, which we refer to as the anomalous behaviour of PyExplainer.
• Figure 5: Our proposed framework for evaluating the reliability and consistency of rule-based XAI techniques. Step 1 includes the dataset selection and pre-processing methods. Step 2 involves the training of ML models, while Step 3 involves the generation of simulated instances. Step 4 investigates the effectiveness of PyExplainer and LIME in generating reliable and consistent explanations for ML models in software analytics tasks.