From Black Boxes to Conversations: Incorporating XAI in a Conversational Agent

TL;DR

The paper addresses making explainable AI accessible to end users by embedding XAI explanations into a conversational agent. It constructs a systematic pipeline that links user questions to explanation methods via an extended XAI question phrase bank, a retrieval mechanism, and template-based natural language generation. Key contributions include a publicly released Question Phrase Bank, a mapping from user intents to XAI methods (e.g., SHAP, DICE, CFProto, Anchors), and a prototype capable of explaining predictions on tabular and image data through natural language templates, along with an evaluation of the NLU component. This framework lays groundwork for scalable, human-centered XAI dialogue and provides data and code to support future development and validation across domains.

Abstract

The goal of Explainable AI (XAI) is to design methods to provide insights into the reasoning process of black-box models, such as deep neural networks, in order to explain them to humans. Social science research states that such explanations should be conversational, similar to human-to-human explanations. In this work, we show how to incorporate XAI in a conversational agent, using a standard design for the agent comprising natural language understanding and generation components. We build upon an XAI question bank, which we extend by quality-controlled paraphrases, to understand the user's information needs. We further systematically survey the literature for suitable explanation methods that provide the information to answer those questions, and present a comprehensive list of suggestions. Our work is the first step towards truly natural conversations about machine learning models with an explanation agent. The comprehensive list of XAI questions and the corresponding explanation methods may support other researchers in providing the necessary information to address users' demands. To facilitate future work, we release our source code and data.

Figures (7)

• Figure 1: XAI user questions are matched to reference questions. Answers are provided by a mapping of reference question to XAI methods along with additional explanations. The X-Agent explains the model's reasoning by feature importance scores of SHAP lundberg_unified_2017 and an explanation how to interpret them.
• Figure 2: Incorporating XAI in conversational agents: 1) Question-Phrase-Generation (QPG) uses a paraphrase generation model on the questions from the XAI question bank liao_questioning_2020. The generated candidates are scored by multiple annotators and ranked, resulting in the XAI question phrase bank. 2) In the Natural Language Understanding (NLU) component, the reference question for a user question is retrieved from the phrase bank. 3) The intent of the reference question defines the XAI method to be applied to the model in the Question-XAI method mapping component (QX). 4) A natural language generation (NLG) component converts the output of the XAI method (e.g., a table, graph, or number) with an answer in natural language. Omitted for overview: data sets are loaded and machine learning models are trained dynamically on user requests.
• Figure 4: Example GPT-3 finetuning, prompt and output to generate XAI paraphrase candidates
• Figure 5: Annotation score distribution for each question category.
• Figure 6: Average human annotation score for all phrase pairs ranked by score. Negative pairs are paraphrases from different questions.