Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auditing Local Explanations is Hard

Robi Bhattacharjee, Ulrike von Luxburg

TL;DR

This work investigates an auditing framework in which a third-party auditor or a collective of users attempts to sanity-check explanations: they can query model decisions and the corresponding local explanations, pool all the information received, and then check for basic consistency properties.

Abstract

In sensitive contexts, providers of machine learning algorithms are increasingly required to give explanations for their algorithms' decisions. However, explanation receivers might not trust the provider, who potentially could output misleading or manipulated explanations. In this work, we investigate an auditing framework in which a third-party auditor or a collective of users attempts to sanity-check explanations: they can query model decisions and the corresponding local explanations, pool all the information received, and then check for basic consistency properties. We prove upper and lower bounds on the amount of queries that are needed for an auditor to succeed within this framework. Our results show that successful auditing requires a potentially exorbitant number of queries -- particularly in high dimensional cases. Our analysis also reveals that a key property is the ``locality'' of the provided explanations -- a quantity that so far has not been paid much attention to in the explainability literature. Looking forward, our results suggest that for complex high-dimensional settings, merely providing a pointwise prediction and explanation could be insufficient, as there is no way for the users to verify that the provided explanations are not completely made-up.

Auditing Local Explanations is Hard

TL;DR

This work investigates an auditing framework in which a third-party auditor or a collective of users attempts to sanity-check explanations: they can query model decisions and the corresponding local explanations, pool all the information received, and then check for basic consistency properties.

Abstract

In sensitive contexts, providers of machine learning algorithms are increasingly required to give explanations for their algorithms' decisions. However, explanation receivers might not trust the provider, who potentially could output misleading or manipulated explanations. In this work, we investigate an auditing framework in which a third-party auditor or a collective of users attempts to sanity-check explanations: they can query model decisions and the corresponding local explanations, pool all the information received, and then check for basic consistency properties. We prove upper and lower bounds on the amount of queries that are needed for an auditor to succeed within this framework. Our results show that successful auditing requires a potentially exorbitant number of queries -- particularly in high dimensional cases. Our analysis also reveals that a key property is the ``locality'' of the provided explanations -- a quantity that so far has not been paid much attention to in the explainability literature. Looking forward, our results suggest that for complex high-dimensional settings, merely providing a pointwise prediction and explanation could be insufficient, as there is no way for the users to verify that the provided explanations are not completely made-up.
Paper Structure (41 sections, 29 theorems, 107 equations, 2 figures, 1 algorithm)

This paper contains 41 sections, 29 theorems, 107 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our contributions: data requirements for auditing.
  3. Related Work
  4. Local Explanations
  5. Preliminaries
  6. Defining local explanations and explainers
  7. Examples of Explainers
  8. Anchors:
  9. Gradient-Based Explanations:
  10. LIME:
  11. A measure of how accurate an explainer is
  12. A measure of how large local regions are
  13. The Auditing Framework
  14. How much data is needed to audit an explainer?
  15. A lower bound on the sample complexity of auditing
  16. ...and 26 more sections

Key Result

Theorem 4.1

Let $\epsilon_1, \epsilon_2 < \frac{1}{48}$ be tolerance parameters, and let $\gamma < \frac{1}{3}$ be any local error threshold. Let $\mu$ be any non-degenerate distribution, and $\lambda > 0$ be any desired level of locality. Then for any auditor $A$ there exists a classifier $f: \mathbb{R}^d \to

Figures (2)

  • Figure 1: Local explanations (see Section \ref{['subsec-explanations']} for notation): a set of data points $x$, their classifications $f(x)$ (red/blue, decision boundary in green), and locally linear explanations $g_x$ at three data points. The explanations $g_x$ are supposed to approximate the behavior of the classifier $f$ within their respective regions $R_x$ (the balls). Panel (a) depicts a regime where there is insufficient data for verifying how accurate the local explanations approximate the classifier $f$. Panel (b) shows explanations that are based on more data (more points in the regions $R_x$), which allows to the audit the explainer.
  • Figure 2: An illustration of Theorem \ref{['thm:high_dimensional_data_is_hard']}, with the concentric blue and red circles depicting the data distribution $\mu$ classified by $f$, and with local explanations being depicted at points A and B. Explanations are forced to either have large local loss (point A) or a low local mass (point B).

Theorems & Definitions (73)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5
  • Definition 2.6
  • Definition 3.1
  • Theorem 4.1: lower bound on the sample complexity of auditing
  • Theorem 4.2: Upper Bound on Sample Complexity of Algorithm \ref{['alg:main']}
  • Theorem 5.1: A high dimensional example
  • ...and 63 more