Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Does Explainable AI Really Mean? A New Conceptualization of Perspectives

Derek Doran, Sarah Schulz, Tarek R. Besold

TL;DR

The paper reframes explainable AI by identifying three cross-field notions—opaque, interpretable, and comprehensible—and proposing a fourth, truly explainable AI that integrates automated reasoning. Through a corpus analysis of major AI venues (2007–2016), it shows how different communities emphasize different explanations, from mechanisms to input–output mappings. It argues that current approaches either reveal inner workings or rely on user-driven interpretation, but do not provide fully automated explanations. The authors advocate neural-symbolic integration with a reasoning engine and domain knowledge to produce end-to-end explanations, aiming to enhance trust, accountability, and ethical use in high-stakes contexts.

Abstract

We characterize three notions of explainable AI that cut across research fields: opaque systems that offer no insight into its algo- rithmic mechanisms; interpretable systems where users can mathemat- ically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing differences in how work on explainable AI is positioned in various fields. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as final step of the generative process.

What Does Explainable AI Really Mean? A New Conceptualization of Perspectives

TL;DR

The paper reframes explainable AI by identifying three cross-field notions—opaque, interpretable, and comprehensible—and proposing a fourth, truly explainable AI that integrates automated reasoning. Through a corpus analysis of major AI venues (2007–2016), it shows how different communities emphasize different explanations, from mechanisms to input–output mappings. It argues that current approaches either reveal inner workings or rely on user-driven interpretation, but do not provide fully automated explanations. The authors advocate neural-symbolic integration with a reasoning engine and domain knowledge to produce end-to-end explanations, aiming to enhance trust, accountability, and ethical use in high-stakes contexts.

Abstract

We characterize three notions of explainable AI that cut across research fields: opaque systems that offer no insight into its algo- rithmic mechanisms; interpretable systems where users can mathemat- ically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing differences in how work on explainable AI is positioned in various fields. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as final step of the generative process.

Paper Structure

This paper contains 4 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Existing Perspectives in Explainable AI
  3. Defining Notions of Explainability
  4. Truly Explainable AI Should Integrate Reasoning

Figures (5)

  • Figure 1: Normalized corpus frequency of "explain" or "explanation".
  • Figure 2: Word clouds of the context of explanation terms in the different proceedings corpora.
  • Figure 3: Relation between opaque, comprehensible, and interpretable AI.
  • Figure 4: External traits of a machine related to explainable AI. The traits depend not only on properties of the learning machine, but also the user. For example, confidence in an interpretable learning system is a function of the user's capability to understand the machine's input/output mapping behavior.
  • Figure 5: Augmenting comprehensible models with a reasoning engine. This engine can combine symbols emitted by a comprehensible machine with a (domain specific) knowledge base encoding relationships between concepts represented by the symbols. The relationships between symbols in the knowledge based can yield a logical deduction about their relationship to the machine's decision.