Towards Human-centered Explainable AI: A Survey of User Studies for Model Explanations

TL;DR

This research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems than in others, but that user evaluations are still rather sparse and incorporate hardly any insights from cognitive or social sciences.

Abstract

Explainable AI (XAI) is widely viewed as a sine qua non for ever-expanding AI research. A better understanding of the needs of XAI users, as well as human-centered evaluations of explainable models are both a necessity and a challenge. In this paper, we explore how HCI and AI researchers conduct user studies in XAI applications based on a systematic literature review. After identifying and thoroughly analyzing 97core papers with human-based XAI evaluations over the past five years, we categorize them along the measured characteristics of explanatory methods, namely trust, understanding, usability, and human-AI collaboration performance. Our research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems than in others, but that user evaluations are still rather sparse and incorporate hardly any insights from cognitive or social sciences. Based on a comprehensive discussion of best practices, i.e., common models, design choices, and measures in user studies, we propose practical guidelines on designing and conducting user studies for XAI researchers and practitioners. Lastly, this survey also highlights several open research directions, particularly linking psychological science and human-centered XAI.

Figures (5)

• Figure 1: Roadmap of our literature analysis. We find out the foundational works of core papers and their application domains using a data-driven method introduced in \ref{['sec:part1']}. Three main research questions in user studies are distilled from core papers. Methods related to measures of each category are discussed in \ref{['sec:part2']}, and findings of the research questions are summarized in \ref{['sec:previous findings']}. Based on the findings, we propose future directions to further promote human-centered XAI in \ref{['sec:summary']}. We distill important messages in this figure, but refer to the discussion in the corresponding sections for more details.
• Figure 2: Distribution of participant numbers in the surveyed user studies by design and participant type (each bar represents one study). Per-design means are indicated in bold.
• Figure 3: Summary cards of the guidelines extracted from past XAI user studies
• Figure 4: Illustration of the foundational research domains (Left): Each dot represents a referenced paper, whose size reflects the number of studied core papers referring to it. Illustration of influenced research domains (Right): Each dot represents a research topic, whose size refers to the number of papers on the same topic. For a clear depiction, only several important research domains are labeled with text. Lines are used to depict reference links, with thicker lines representing a greater number of links. Core paper categories are in blue (Middle). Circles are used to indicate a hierarchical structure of keywords.
• Figure 5: Chronology of commonly used XAI methods from reviewed papers.