People Attribute Purpose to Autonomous Vehicles When Explaining Their Behavior: Insights from Cognitive Science for Explainable AI

TL;DR

This study investigates how explanatory modes from cognitive science shape laypeople's understanding of autonomous vehicle behavior. It introduces a framework distinguishing teleological, mechanistic, counterfactual, and descriptive explanations and tests it in a two-study AV setting, culminating in the HEADD dataset of human-generated explanations and evaluations. Results show teleological explanations are rated as more satisfying and quality/trust are strongly predicted by perceived teleology, independent of whether the explainee is seen as autonomous or human. The work provides principled guidance for designing human-centered XAI in safety-critical autonomous systems and distributes the HEADD resource for further research.

Abstract

It is often argued that effective human-centered explainable artificial intelligence (XAI) should resemble human reasoning. However, empirical investigations of how concepts from cognitive science can aid the design of XAI are lacking. Based on insights from cognitive science, we propose a framework of explanatory modes to analyze how people frame explanations, whether mechanistic, teleological, or counterfactual. Using the complex safety-critical domain of autonomous driving, we conduct an experiment consisting of two studies on (i) how people explain the behavior of a vehicle in 14 unique scenarios (N1=54) and (ii) how they perceive these explanations (N2=382), curating the novel Human Explanations for Autonomous Driving Decisions (HEADD) dataset. Our main finding is that participants deem teleological explanations significantly better quality than counterfactual ones, with perceived teleology being the best predictor of perceived quality. Based on our results, we argue that explanatory modes are an important axis of analysis when designing and evaluating XAI and highlight the need for a principled and empirically grounded understanding of the cognitive mechanisms of explanation. The HEADD dataset and our code are available at: https://datashare.ed.ac.uk/handle/10283/8930.

Figures (10)

• Figure 1: The workflow of our experiment with two studies. In the first study (\ref{['ssec:method:1']}), online participants generate explanations along the four explanatory modes. In the second study (\ref{['ssec:method:2']}), new participants evaluate the generated explanations along various subjective measures. Finally, we quantitatively analyze participant judgments and test our hypotheses (\ref{['sec:quant-results']}).
• Figure 2: Three example scenarios. Participants were always asked to explain the behavior of the blue ego vehicle. (Left; #8). The blue car slows down before turning right, as its view is blocked by a building. Once the view is clear, the blue car notices pedestrians at the crossing and stops. (Mid; #11). The blue car is passing a row of parked cars when it perceives a ball rolling onto the road. It sharply breaks, as a child emerges from behind a truck. (Right; #12) The blue car waits behind a truck obscuring its vision of the road. It keeps on waiting as the truck passes between the parked cars, to potentially avoid other oncoming cars.
• Figure 3: The workflow of Study 1. The text of each question (highlighted in italics) is copied verbatim as it appeared in the experiment to participants, with words in braces replaced according to the independent variables. Red boxes required participants to read and understand instructions. Blue boxes required input from the participants.
• Figure 4: The workflow of Study 2. The text of each question (highlighted in italic) is verbatim as it appeared in the experiment. Red boxes required participants to read and understand instructions. Blue boxes required input from the participants.
• Figure 5: Zero-order correlation between ratings of Study 2. Correlation coefficients circled in orange are non-significant after Bonferroni correction. The Y-axis labels are the dependent variables of Study 2 (cf., \ref{['sssec:method:2:dependent']}); The most correlated variables are: (i) Satisfying: whether the participants found the explanation satisfying; (ii) Complete: perceived level of explanation completeness; (iii) SufficientDetail: perceived level of detail in the explanation.