Conversational Explanations: Discussing Explainable AI with Non-AI Experts

TL;DR

This work tackles the scarcity of data for building conversational XAI systems by training a vision-language-based explanation engine (EMCEE) on self-generated synthetic conversations. It introduces a repetition penalty to increase data diversity and a hallucination detector to filter incorrect turns, implementing a multi-round data-generation–finetuning loop that progressively improves explanations for non-AI experts. Automatic and human evaluations show EMCEE markedly outperforms a strong baseline in both linguistic alignment (BLEU/ROUGE) and user-centered outcomes like comprehension, acceptance, trust, and collaborative use of static explanations, with qualitative analyses confirming higher truthfulness and understandability. The results demonstrate the practicality and impact of conversational explanations that can answer free-form follow-up questions after static explanations, advancing human-AI collaboration in high-stakes domains.

Abstract

Explainable AI (XAI) aims to provide insights into the decisions made by AI models. To date, most XAI approaches provide only one-time, static explanations, which cannot cater to users' diverse knowledge levels and information needs. Conversational explanations have been proposed as an effective method to customize XAI explanations. However, building conversational explanation systems is hindered by the scarcity of training data. Training with synthetic data faces two main challenges: lack of data diversity and hallucination in the generated data. To alleviate these issues, we introduce a repetition penalty to promote data diversity and exploit a hallucination detector to filter out untruthful synthetic conversation turns. We conducted both automatic and human evaluations on the proposed system, fEw-shot Multi-round ConvErsational Explanation (EMCEE). For automatic evaluation, EMCEE achieves relative improvements of 81.6% in BLEU and 80.5% in ROUGE compared to the baselines. EMCEE also mitigates the degeneration of data quality caused by training on synthetic data. In human evaluations (N=60), EMCEE outperforms baseline models and the control group in improving users' comprehension, acceptance, trust, and collaboration with static explanations by large margins. Through a fine-grained analysis of model responses, we further demonstrate that training on self-generated synthetic data improves the model's ability to generate more truthful and understandable answers, leading to better user interactions. To the best of our knowledge, this is the first conversational explanation method that can answer free-form user questions following static explanations.

Figures (18)

• Figure 1: The Overall Workflow of EMCEE. $V_i$ denotes the VLM and $D_i$ denotes the synthetic conversation data in the $i$-th iteration. Starting from a pretrained VLM $V_1$, we first generate diverse synthetic conversations $D_1$ with the repetition penalty. Next, we use a hallucination detector to clean synthetic data, producing cleaned data $D_1^{\text{\,clean}}$. We then finetune the VLM on $D_1^{\text{\,clean}}$, which creates $V_2$, and this process repeats.
• Figure 2: The interface where users discuss static explanations with a conversational agent. Part A: Information about static explanations, including a task description, a description of the prediction model, a model input, a model output, an explanation generated by the explanation model, and a description of the explanation. Part B: A chatbox where users converse with a conversational agent to clarify the explanation.
• Figure 3: BLEU-4 and Rouge-L scores over the number of training iterations for LLaVa-1.5, EMCEE and different ablated version of EMCEE.
• Figure 4: Model selection accuracy for (a) LIME and (b) Grad-CAM (c) Integrated Gradients (d) SHAP before and after conditions.
• Figure 5: Subjective understanding score for (a) LIME and (b) Grad-CAM (c) Integrated Gradients (d) SHAP before and after conditions.