Information That Matters: Exploring Information Needs of People Affected by Algorithmic Decisions

TL;DR

The XAI Novice Question Bank is presented, an extension of the XAI Question Bank containing a catalog of information needs from AI novices in two use cases: employment prediction and health monitoring, and aims to support the inclusion of AI novices in explainability efforts by highlighting their information needs, aims, and challenges.

Abstract

Every AI system that makes decisions about people has a group of stakeholders that are personally affected by these decisions. However, explanations of AI systems rarely address the information needs of this stakeholder group, who often are AI novices. This creates a gap between conveyed information and information that matters to those who are impacted by the system's decisions, such as domain experts and decision subjects. To address this, we present the "XAI Novice Question Bank," an extension of the XAI Question Bank containing a catalog of information needs from AI novices in two use cases: employment prediction and health monitoring. The catalog covers the categories of data, system context, system usage, and system specifications. We gathered information needs through task-based interviews where participants asked questions about two AI systems to decide on their adoption and received verbal explanations in response. Our analysis showed that participants' confidence increased after receiving explanations but that their understanding faced challenges. These included difficulties in locating information and in assessing their own understanding, as well as attempts to outsource understanding. Additionally, participants' prior perceptions of the systems' risks and benefits influenced their information needs. Participants who perceived high risks sought explanations about the intentions behind a system's deployment, while those who perceived low risks rather asked about the system's operation. Our work aims to support the inclusion of AI novices in explainability efforts by highlighting their information needs, aims, and challenges. We summarize our findings as five key implications that can inform the design of future explanations for lay stakeholder audiences.

Figures (4)

• Figure 1: Depiction of the study procedure. Participants received a short introduction to the study, filled out a questionnaire on demographic information and prior knowledge, and then were presented with one of two use cases. Participants could then ask questions about the given use case in two 15-minute inquiry phases. In the second of these phases, they received the XAI Question Bank liao2020 for reference. They then voted on adopting the discussed system and answered three interview questions. Participants were asked two times for their self-reported understanding, perceived decision confidence, and perceived risks and benefits of the system: before the inquiry phases and after making the decision.
• Figure 2: Interview transcripts showing examples of question-driven explanations. The study examiner responded to participants' questions with verbal explanations during the inquiry phases I and II. All explanations were based on publicly available information about the systems and were phrased so as not to convey any personal opinion or judgment.
• Figure 3: The XAI Novice Question Bank and system-inquiry diagram. Depicted above are four categories of questions that subsume participants' inquiries about the two ADM use cases (Section \ref{['sec:use_cases']}). An asterisk (*) indicates that the question is already present in the XAI Question Bank liao2020. Numbers and letters are added to the side of each question to refer to stakeholders (pictograms) and procedures (arrows) in the system deployment process shown below. System and user are framed together to indicate this part of deployment as the core interaction with the system. In the system section, the arrow indicates the separation of data and model. The two-way arrows in procedure C depict interactions between the user and the system, and the twisted arrows in procedures D and E indicate the complex effects that system deployment and usage have on both the target group and society.
• Figure 4: Changes in perceived risks and benefits. Participants were asked two times for their perceptions of the given ADM system's risks and benefits for both society and them personally (as described in Section \ref{['sec:method']}) on scales from 0 (low) to 10 (high). Lines depict perception changes from before (gray) to after (colored) the inquiry phases, split between the employment prediction use case (A) above and the health wristband use case (B) below. While perceptions changed drastically in A and are distributed throughout all quadrants, perceptions in B changed comparatively little and remained mostly in the upper left quadrant.