Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisions

Somayeh Molaei, Lionel P. Robert, Nikola Banovic

TL;DR

This work investigates end-user needs for explanations of AI-driven autonomous vehicles and evaluates whether interactive, text-based explanations improve understanding and AI literacy. Through a formative Wizard-of-Oz study (Study 1) and a subsequent quantitative, LLM-enabled experiment (Study 2), the authors delineate end-user information needs across situational awareness, hypothesis testing, decision contestation, trust repair, and general AV capabilities. They find that static explanations paired with follow-up Q&A significantly enhances scenario understanding and AI literacy, while purely interactive Q&A yields strong benefits mainly for scenario-specific questions; the combination static+Q&A provides the strongest overall impact. The findings advocate for holistic, dialogue-based HCXAI designs that address multi-subsystem AVs and emphasize socio-technical context, while leveraging LLMs carefully to avoid hallucinations and ensure reliable end-user education and trust.

Abstract

Improving end-users' understanding of decisions made by autonomous vehicles (AVs) driven by artificial intelligence (AI) can improve utilization and acceptance of AVs. However, current explanation mechanisms primarily help AI researchers and engineers in debugging and monitoring their AI systems, and may not address the specific questions of end-users, such as passengers, about AVs in various scenarios. In this paper, we conducted two user studies to investigate questions that potential AV passengers might pose while riding in an AV and evaluate how well answers to those questions improve their understanding of AI-driven AV decisions. Our initial formative study identified a range of questions about AI in autonomous driving that existing explanation mechanisms do not readily address. Our second study demonstrated that interactive text-based explanations effectively improved participants' comprehension of AV decisions compared to simply observing AV decisions. These findings inform the design of interactions that motivate end-users to engage with and inquire about the reasoning behind AI-driven AV decisions.

What Do People Want to Know About Artificial Intelligence (AI)? The Importance of Answering End-User Questions to Explain Autonomous Vehicle (AV) Decisions

TL;DR

This work investigates end-user needs for explanations of AI-driven autonomous vehicles and evaluates whether interactive, text-based explanations improve understanding and AI literacy. Through a formative Wizard-of-Oz study (Study 1) and a subsequent quantitative, LLM-enabled experiment (Study 2), the authors delineate end-user information needs across situational awareness, hypothesis testing, decision contestation, trust repair, and general AV capabilities. They find that static explanations paired with follow-up Q&A significantly enhances scenario understanding and AI literacy, while purely interactive Q&A yields strong benefits mainly for scenario-specific questions; the combination static+Q&A provides the strongest overall impact. The findings advocate for holistic, dialogue-based HCXAI designs that address multi-subsystem AVs and emphasize socio-technical context, while leveraging LLMs carefully to avoid hallucinations and ensure reliable end-user education and trust.

Abstract

Improving end-users' understanding of decisions made by autonomous vehicles (AVs) driven by artificial intelligence (AI) can improve utilization and acceptance of AVs. However, current explanation mechanisms primarily help AI researchers and engineers in debugging and monitoring their AI systems, and may not address the specific questions of end-users, such as passengers, about AVs in various scenarios. In this paper, we conducted two user studies to investigate questions that potential AV passengers might pose while riding in an AV and evaluate how well answers to those questions improve their understanding of AI-driven AV decisions. Our initial formative study identified a range of questions about AI in autonomous driving that existing explanation mechanisms do not readily address. Our second study demonstrated that interactive text-based explanations effectively improved participants' comprehension of AV decisions compared to simply observing AV decisions. These findings inform the design of interactions that motivate end-users to engage with and inquire about the reasoning behind AI-driven AV decisions.
Paper Structure (34 sections, 6 figures, 4 tables)

This paper contains 34 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Study 1: What Do End-Users Seek Explanations About?
  4. Method
  5. Driving Scenario Videos and Metadata
  6. User Study Software Implementation
  7. Tasks and Procedures
  8. Participants
  9. Results
  10. Understanding the AV Context and Status (15.5 % of questions)
  11. Hypothesizing Behaviors & Diagnosing Events (18.5% of questions)
  12. Contesting AV Decisions (18.5% of questions)
  13. Repairing Trust (23.9% of questions)
  14. What can AV do? How does it work? (23.3% of questions)
  15. Takeaways
  16. ...and 19 more sections

Figures (6)

  • Figure 1: Wizard-of-Oz Design Probe User Interface showing: a) 30-second-long driving scenario video recording from inside of a vehicle with the perspective of looking forward and out the windshield, b) task comprehension quiz, c) participant-facing conversational XAI interface, and d) wizard-facing conversational XAI interface.
  • Figure 2: Prototype conversational XAI user interface showing: a) static natural language explanation for a driving scenario, and b) LLM-powered participant-facing conversational XAI Q&A interface.
  • Figure 3: Participant scores across the four conditions: a) task expertise score (i.e., driving scenario understanding), b) AI literacy score, and c) NASA-TLX workload.
  • Figure 4: Percentage of participants who correctly answered each AI literacy assessment question.
  • Figure 5: The frequency of driving scenarios (y-axis) per number of questions that the participants asked in $Q\&A$ and $static+Q\&A$ conditions (x-axis).
  • ...and 1 more figures