Is Conversational XAI All You Need? Human-AI Decision Making With a Conversational XAI Assistant

TL;DR

This paper examines whether a Conversational XAI (CXAI) assistant enhances user understanding, trust, and appropriate reliance more than an XAI dashboard in loan-approval decisions. It compares rule-based CXAI, evaluative CXAI, and an LLM-powered CXAI variant, revealing modest gains in understanding and trust but a general trend toward over-reliance across interactive interfaces. Notably, LLM-powered CXAI amplifies over-reliance and can degrade objective understanding, suggesting that higher conversational plausibility may foster an illusion of explanatory depth rather than calibrated judgment. The findings underscore the need for seamful, trustworthy CXAI designs that engage users without encouraging uncritical reliance on AI advice.

Abstract

Explainable artificial intelligence (XAI) methods are being proposed to help interpret and understand how AI systems reach specific predictions. Inspired by prior work on conversational user interfaces, we argue that augmenting existing XAI methods with conversational user interfaces can increase user engagement and boost user understanding of the AI system. In this paper, we explored the impact of a conversational XAI interface on users' understanding of the AI system, their trust, and reliance on the AI system. In comparison to an XAI dashboard, we found that the conversational XAI interface can bring about a better understanding of the AI system among users and higher user trust. However, users of both the XAI dashboard and conversational XAI interfaces showed clear overreliance on the AI system. Enhanced conversations powered by large language model (LLM) agents amplified over-reliance. Based on our findings, we reason that the potential cause of such overreliance is the illusion of explanatory depth that is concomitant with both XAI interfaces. Our findings have important implications for designing effective conversational XAI interfaces to facilitate appropriate reliance and improve human-AI collaboration. Code can be found at https://github.com/delftcrowd/IUI2025_ConvXAI

Figures (6)

• Figure 1: Screenshot of the loan approval task interface. This is the first stage of decision making. (A) Loan Applicant profile is shown in the table with 11 features. (B) To help understand the tabular data, we also provided a textual description below. (C) After going through the profile, participants are asked to decide whether this loan application is 'Credit Worthy' or 'Not Credit Worthy.'
• Figure 2: Screenshots illustrating the XAI interfaces we designed. Additional screenshots demonstrating all XAI methods across both XAI interfaces are available in the supplementary materials.
• Figure 3: Illustration of the procedure that participants followed in our study. This flow chart describes the experimental condition CXAI.
• Figure 4: Bar plot illustrating the explanation utility across conditions. Error bars represent the $95\%$ confidence interval.
• Figure 5: Line plot illustrating the confidence dynamics among users after receiving the AI advice (and explanations). The orange line and blue line illustrate the confidence dynamics before and after receiving AI advice (and explanations), respectively.