The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective

TL;DR

This work formalizes the disagreement problem in explainable ML, showing that state-of-the-art post hoc explanations often disagree across tabular, text, and image data. It combines practitioner-informed definitions with a rigorous quantitative framework of six metrics to measure top-k and features-of-interest disagreements, followed by extensive empirical analyses over four datasets, six explanation methods, and six predictive models. An online user study reveals that practitioners frequently rely on ad hoc heuristics when resolving disagreements, highlighting a gap between theory and practice. The findings motivate developing principled evaluation frameworks and guidance for selecting among explanation techniques in high-stakes settings.

Abstract

As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of whether and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we formalize and study the disagreement problem in explainable machine learning. More specifically, we define the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how practitioners resolve these disagreements. We first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and six different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that (1) state-of-the-art explanation methods often disagree in terms of the explanations they output, and (2) machine learning practitioners often employ ad hoc heuristics when resolving such disagreements. These findings suggest that practitioners may be relying on misleading explanations when making consequential decisions. They also underscore the importance of developing principled frameworks for effectively evaluating and comparing explanations output by various explanation techniques.

Figures (21)

• Figure 1: Disagreement between explanation methods for neural network model trained on COMPAS dataset measured by six metrics: rank correlation and pairwise rank agreement across all features, and feature, rank, sign, and signed rank agreement across top $k=5$ features. Heatmaps show the average metric value over the test set data points for each pair of explanation methods, with lower values (lighter colors) indicating stronger disagreement. The disagreement between a pair of explanations is considered non-trivial if less than 75% of the features agree (i.e., if feature agreement, rank agreement, sign agreement, signed rank agreement, or pairwise rank agreement is less than 0.75), or if the correlation among features is only moderately positive, nonexistent, or negative (i.e. if the rank correlation is less than 0.50). Across all six heatmaps, the standard error ranges between 0 and 0.01.
• Figure 2: Disagreement between explanation methods for neural network model trained on COMPAS dataset measured by rank agreement (top row) and signed rank agreement (bottom row) at top-$k$ features for increasing values of $k$. Each cell in the heatmap shows the metric value averaged over test set data points for each pair of explanation methods, with lower values (lighter colors) indicating stronger disagreement. We follow the same disagreement interpretation described in Figure \ref{['mainfig-compas-constantk']}. Across all six heatmaps, the standard error ranges between 0 and 0.01.
• Figure 3: Distribution of rank correlation over all features for neural network model trained on COMPAS (left), and rank correlation across all features (middle) and signed rank agreement across top-$11$ features (right) for neural network model trained on AG_News.
• Figure 4: Disagreement between explanation methods for LSTM trained on the AG_News dataset using $k=11$ features for metrics operating on top-$k$ features, and all features for other metrics. Each heatmap shows the metric value averaged over test data for each pair of explanation methods. Lighter colors indicate more disagreement. Standard error ranges from 0.0 to 0.0025 for all six metrics.
• Figure 5: The user interface for a prompt. The user is shown two explanations for a COMPAS data point, showing the feature importance value of each of the 7 features. Red and blue indicate negative and positive feature values, respectively. See the text for more details.

Theorems & Definitions (1)

• Remark 1