Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Model Explanation to Data Misinterpretation: Uncovering the Pitfalls of Post Hoc Explainers in Business Research

Ronilo Ragodos, Tong Wang, Lu Feng, Yu, Hu

TL;DR

It is often not appropriate to infer data insights from post hoc explanations of machine learning models, and appropriate alternative uses are articulate to facilitate the proposition and subsequent empirical investigation of hypotheses.

Abstract

Machine learning models have been increasingly used in business research. However, most state-of-the-art machine learning models, such as deep neural networks and XGBoost, are black boxes in nature. Therefore, post hoc explainers that provide explanations for machine learning models by, for example, estimating numerical importance of the input features, have been gaining wide usage. Despite the intended use of post hoc explainers being explaining machine learning models, we found a growing trend in business research where post hoc explanations are used to draw inferences about the data. In this work, we investigate the validity of such use. Specifically, we investigate with extensive experiments whether the explanations obtained by the two most popular post hoc explainers, SHAP and LIME, provide correct information about the true marginal effects of X on Y in the data, which we call data-alignment. We then identify what factors influence the alignment of explanations. Finally, we propose a set of mitigation strategies to improve the data-alignment of explanations and demonstrate their effectiveness with real-world data in an econometric context. In spite of this effort, we nevertheless conclude that it is often not appropriate to infer data insights from post hoc explanations. We articulate appropriate alternative uses, the most important of which is to facilitate the proposition and subsequent empirical investigation of hypotheses. The ultimate goal of this paper is to caution business researchers against translating post hoc explanations of machine learning models into potentially false insights and understanding of data.

From Model Explanation to Data Misinterpretation: Uncovering the Pitfalls of Post Hoc Explainers in Business Research

TL;DR

It is often not appropriate to infer data insights from post hoc explanations of machine learning models, and appropriate alternative uses are articulate to facilitate the proposition and subsequent empirical investigation of hypotheses.

Abstract

Machine learning models have been increasingly used in business research. However, most state-of-the-art machine learning models, such as deep neural networks and XGBoost, are black boxes in nature. Therefore, post hoc explainers that provide explanations for machine learning models by, for example, estimating numerical importance of the input features, have been gaining wide usage. Despite the intended use of post hoc explainers being explaining machine learning models, we found a growing trend in business research where post hoc explanations are used to draw inferences about the data. In this work, we investigate the validity of such use. Specifically, we investigate with extensive experiments whether the explanations obtained by the two most popular post hoc explainers, SHAP and LIME, provide correct information about the true marginal effects of X on Y in the data, which we call data-alignment. We then identify what factors influence the alignment of explanations. Finally, we propose a set of mitigation strategies to improve the data-alignment of explanations and demonstrate their effectiveness with real-world data in an econometric context. In spite of this effort, we nevertheless conclude that it is often not appropriate to infer data insights from post hoc explanations. We articulate appropriate alternative uses, the most important of which is to facilitate the proposition and subsequent empirical investigation of hypotheses. The ultimate goal of this paper is to caution business researchers against translating post hoc explanations of machine learning models into potentially false insights and understanding of data.
Paper Structure (49 sections, 3 theorems, 24 equations, 22 figures, 4 tables)

This paper contains 49 sections, 3 theorems, 24 equations, 22 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Post Hoc Explainers - SHAP and LIME
  4. The Evaluation of Post hoc Explanations
  5. Use of SHAP and LIME in Business Research
  6. Explanation Pipeline
  7. Data Generation Process
  8. Applying the Explanation Pipeline
  9. Misalignment of Post Hoc Explanations
  10. New Metrics for Assessing Post hoc Explanations
  11. Pervasiveness of Data-Misalignment - a Large Scale Evaluation
  12. Contributing Factors to Misalignment
  13. Explainers May Inherently Be Misaligned
  14. $e_{\text{SHAP}}$ when Explaining a Linear Model
  15. $e_{\text{LIME}}$ when Explaining a Linear Model
  16. ...and 34 more sections

Key Result

Theorem 1

Suppose $f:\mathbb{R}^D\rightarrow \mathbb{R}$ that maps $\mathbf{x} \mapsto \boldsymbol{\beta}^T \mathbf{x}$ is explained at a sample $\boldsymbol{\xi}$ drawn from $\mathcal{N}(\mathbf{0}_D, \mathbf{I}_{D\times D})$ with LIME. Suppose further that LIME does not discretize features and samples i.i.d

Figures (22)

  • Figure 1: Explanation Pipeline: the pipeline of using post hoc explainers to explain a machine learning model. Additionally inferring information about the actual marginal effects of the features constitutes mistaken usage of the pipeline.
  • Figure 2: Year-by-year breakdown of number of papers citing either LIME or SHAP that belong to business-adjacent Web of Science categories.
  • Figure 3: John's demographics along with a plot of the SHAP values obtained by explaining $\mathcal{M}$'s prediction for John, compared against the ground truth from $\mathcal{G}$.
  • Figure 4: Illustration of the three aspects of data-alignment evaluation. For plot (c) each point in the line plot represents a mean relevance for a given $k$. The error bars represent the standard error. The total number of features is 10.
  • Figure 5: Relationship of number of features $D$ with feature-wise mean directionality, concordance, and relevance for $k=3$
  • ...and 17 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Remark 1
  • Lemma 2
  • Lemma 3
  • Remark 2: Connection to condition numbers