The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR

TL;DR

The paper investigates what GDPR driven legal experts expect from explanations of automated decisions and whether current XAI methods meet those expectations. It presents Explanation Dialogues, an expert focus study combining an online questionnaire and follow-up interviews in a credit domain use-case, analyzed with grounded theory. Findings show that state of the art explanations are often hard to understand and information deficient, with mixed perceptions of GDPR compliance that hinge on how explanations enable exercise of rights. The work offers a hierarchical codes framework, practitioner-oriented recommendations for designing explanations, and legal pointers on contestability, transparency thresholds, and party relations, highlighting the need for interdisciplinary, user centered design. Overall, the study informs both tool developers and policymakers on how to align XAI outputs with legal rights and industry practices in Europe.

Abstract

Explainable AI (XAI) provides methods to understand non-interpretable machine learning models. However, we have little knowledge about what legal experts expect from these explanations, including their legal compliance with, and value against European Union legislation. To close this gap, we present the Explanation Dialogues, an expert focus study to uncover the expectations, reasoning, and understanding of legal experts and practitioners towards XAI, with a specific focus on the European General Data Protection Regulation. The study consists of an online questionnaire and follow-up interviews, and is centered around a use-case in the credit domain. We extract both a set of hierarchical and interconnected codes using grounded theory, and present the standpoints of the participating experts towards XAI. We find that the presented explanations are hard to understand and lack information, and discuss issues that can arise from the different interests of the data controller and subject. Finally, we present a set of recommendations for developers of XAI methods, and indications of legal areas of discussion. Among others, recommendations address the presentation, choice, and content of an explanation, technical risks as well as the end-user, while we provide legal pointers to the contestability of explanations, transparency thresholds, intellectual property rights as well as the relationship between involved parties.

Figures (8)

• Figure 1: Online questionnaire structure: each questionnaire consists the Case TP and TN. Further, it contains three different explanation types, one of which is global, two of which are local. Each explanation is followed by a set of questions and each case by a set of comparison questions. The questionnaire starts with an introduction and general information about the ADM system and closes with a conclusion and a set of closing questions. The specific explanations and the order of cases are sampled randomly.
• Figure 2: Main steps in the evaluation process.
• Figure 3: Exemplary results from the multiple choice questions. Left: answers to "Judge how well you understand the decision based on the shown explanation on the following scale". Right: answers to "The explanation provided does not pose a [potential] conflict with the interest and rights of the bank (e.g., intellectual property)".
• Figure 4: Hierarchy of codes. Dark gray boxes refer to core phenomena, gray boxes to themes, white boxes to sub-themes and light gray boxes to categories (top to bottom).
• Figure 5: SHAP explanations. Upper: global SHAP, middle and lower: local SHAP for Case TP local (middle panel w.r.t. a low credit risk score/good credit, lower panel w.r.t. a high credit risk score/bad credit).