Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

False Sense of Security in Explainable Artificial Intelligence (XAI)

Neo Christopher Chung, Hongkyou Chung, Hearim Lee, Lennart Brocki, Hongbeom Chung, George Dyer

TL;DR

The paper analyzes the gap between policy rhetoric demanding explainability in AI and the technical reality that current XAI methods provide incomplete, unstable explanations. It reviews US and EU regulatory frameworks (EO, AI Act, GDPR, AI Liability Directive) through a right-to-explanation lens, highlighting ambiguous or evolving obligations. It identifies five critical failure modes of XAI (robustness, adversarial manipulation, partial explanations, drift, and anthropomorphization) and discusses market dynamics that promote superficial explanations. The authors argue that without precise, technically informed standards, governance risks becoming a false sense of security and call for careful, ongoing alignment between regulation, industry practice, and rigorous evaluation.

Abstract

A cautious interpretation of AI regulations and policy in the EU and the USA place explainability as a central deliverable of compliant AI systems. However, from a technical perspective, explainable AI (XAI) remains an elusive and complex target where even state of the art methods often reach erroneous, misleading, and incomplete explanations. "Explainability" has multiple meanings which are often used interchangeably, and there are an even greater number of XAI methods - none of which presents a clear edge. Indeed, there are multiple failure modes for each XAI method, which require application-specific development and continuous evaluation. In this paper, we analyze legislative and policy developments in the United States and the European Union, such as the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, the AI Act, the AI Liability Directive, and the General Data Protection Regulation (GDPR) from a right to explanation perspective. We argue that these AI regulations and current market conditions threaten effective AI governance and safety because the objective of trustworthy, accountable, and transparent AI is intrinsically linked to the questionable ability of AI operators to provide meaningful explanations. Unless governments explicitly tackle the issue of explainability through clear legislative and policy statements that take into account technical realities, AI governance risks becoming a vacuous "box-ticking" exercise where scientific standards are replaced with legalistic thresholds, providing only a false sense of security in XAI.

False Sense of Security in Explainable Artificial Intelligence (XAI)

TL;DR

The paper analyzes the gap between policy rhetoric demanding explainability in AI and the technical reality that current XAI methods provide incomplete, unstable explanations. It reviews US and EU regulatory frameworks (EO, AI Act, GDPR, AI Liability Directive) through a right-to-explanation lens, highlighting ambiguous or evolving obligations. It identifies five critical failure modes of XAI (robustness, adversarial manipulation, partial explanations, drift, and anthropomorphization) and discusses market dynamics that promote superficial explanations. The authors argue that without precise, technically informed standards, governance risks becoming a false sense of security and call for careful, ongoing alignment between regulation, industry practice, and rigorous evaluation.

Abstract

A cautious interpretation of AI regulations and policy in the EU and the USA place explainability as a central deliverable of compliant AI systems. However, from a technical perspective, explainable AI (XAI) remains an elusive and complex target where even state of the art methods often reach erroneous, misleading, and incomplete explanations. "Explainability" has multiple meanings which are often used interchangeably, and there are an even greater number of XAI methods - none of which presents a clear edge. Indeed, there are multiple failure modes for each XAI method, which require application-specific development and continuous evaluation. In this paper, we analyze legislative and policy developments in the United States and the European Union, such as the Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, the AI Act, the AI Liability Directive, and the General Data Protection Regulation (GDPR) from a right to explanation perspective. We argue that these AI regulations and current market conditions threaten effective AI governance and safety because the objective of trustworthy, accountable, and transparent AI is intrinsically linked to the questionable ability of AI operators to provide meaningful explanations. Unless governments explicitly tackle the issue of explainability through clear legislative and policy statements that take into account technical realities, AI governance risks becoming a vacuous "box-ticking" exercise where scientific standards are replaced with legalistic thresholds, providing only a false sense of security in XAI.
Paper Structure (16 sections, 4 figures)

This paper contains 16 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Legislative and Policy Developments
  3. United States of America
  4. European Union
  5. AI Act
  6. General Data Protection Regulation (GDPR)
  7. AI Liability Directive
  8. Polysemy of "Explainability"
  9. Failure Modes of XAI
  10. Robustness
  11. Adversarial Attacks
  12. Partial Explanations
  13. Data and Concept Drift
  14. Anthropomorphization
  15. Market Conditions
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Explosive growth of XAI research. The number of XAI papers in 12 AI/ML conferences are counted and categorized by Nauta et al., 2023.
  • Figure 2: The first row shows the same image of 3, with different noise. CNN trained on MNIST correctly classifies these sample images as '3'. Explanations from LIME are shown in the second row.
  • Figure 3: ViT model is fine-tuned on lung CT scans, to classify benign vs. malignant tumors. Attention maps are popular explanations for ViT.
  • Figure 4: (a) The training data is initially used to produce a model, whose decision boundary classifies data in production. (b) Data drift occurs when the distribution of data has significantly changed. (c) The relationship between data and concept (i.e., class) may change over time, resulting in a concept drift. (d) Concept drift could arise from introduction of a new class, that was not included in the training data.