May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability

TL;DR

This paper investigates whether free-form conversational explanations can enhance understanding and trust in static XAI explanations. Using a Wizard-of-Oz design with 120 participants, it compares conversational sessions with reading sessions across Grad-CAM and LIME explanations, measuring objective model-selection accuracy and 13-item subjective judgments. Results show that conversations significantly improve both objective comprehension and subjective measures (usefulness, ease-of-use, behavioral intention, and trust), with deeper gains when using Grad-CAM explanations. The findings support designing conversational AI explainability systems that tailor explanations to users’ knowledge and information needs, and they offer practical guidance for future dialogue-enabled XAI tools, despite limitations such as reliance on static explanations and a constrained participant pool.

Abstract

Research in explainable AI (XAI) aims to provide insights into the decision-making process of opaque AI models. To date, most XAI methods offer one-off and static explanations, which cannot cater to the diverse backgrounds and understanding levels of users. With this paper, we investigate if free-form conversations can enhance users' comprehension of static explanations, improve acceptance and trust in the explanation methods, and facilitate human-AI collaboration. Participants are presented with static explanations, followed by a conversation with a human expert regarding the explanations. We measure the effect of the conversation on participants' ability to choose, from three machine learning models, the most accurate one based on explanations and their self-reported comprehension, acceptance, and trust. Empirical results show that conversations significantly improve comprehension, acceptance, trust, and collaboration. Our findings highlight the importance of customized model explanations in the format of free-form conversations and provide insights for the future design of conversational explanations.

Figures (15)

• Figure 1: Example explanations generated by Grad-CAM and LIME. (a) is the input to the classification model (Swin Transformer), (b) is the explanation generated by Grad-CAM, and (c) is the explanation generated by LIME. The predicted class of the model is "Siamese cat".
• Figure 2: An example of the objective evaluation. The objective evaluation aims to objectively measure participants' comprehension of static explanations. Each choice contains a prediction from a different classification model, paired with its respective static explanation. Participants need to choose the best model based on the explanations.
• Figure 3: The web page where users can discuss static explanations with an expert.
• Figure 4: Objective decision accuracy for different groups before and after conditions.
• Figure 5: Subjective understanding score for (a) LIME and (b) Grad-CAM before and after conditions.