Show Me How: Benefits and Challenges of Agent-Augmented Counterfactual Explanations for Non-Expert Users

TL;DR

Counterfactual explanations improve decision-making but risk impracticality for non-experts. The authors propose agent-augmented counterfactuals via a healthcare chatbot to provide context-aware, actionable recommendations for patients at high cardiovascular risk, grounded in LLM-based conversational agents with safeguards. A general guideline set is developed and instantiated in a CVD-risk tool, evaluated in a mixed-methods study with 34 participants, showing higher objective actionability for informed users and potential novice over-reliance. The contributions include theoretical guidelines, a practical open-source artifact, and empirical insights to design safe, contextually grounded XAI for non-experts, with implications for broader deployment beyond healthcare.

Abstract

Counterfactual explanations offer actionable insights by illustrating how changes to inputs can lead to different outcomes. However, these explanations often suffer from ambiguity and impracticality, limiting their utility for non-expert users with limited AI knowledge. Augmenting counterfactual explanations with Large Language Models (LLMs) has been proposed as a solution, but little research has examined their benefits and challenges for non-experts. To address this gap, we developed a healthcare-focused system that leverages conversational AI agents to enhance counterfactual explanations, offering clear, actionable recommendations to help patients at high risk of cardiovascular disease (CVD) reduce their risk. Evaluated through a mixed-methods study with 34 participants, our findings highlight the effectiveness of agent-augmented counterfactuals in improving actionable recommendations. Results further indicate that users with prior experience using conversational AI demonstrated greater effectiveness in utilising these explanations compared to novices. Furthermore, this paper introduces a set of generic guidelines for creating augmented counterfactual explanations, incorporating safeguards to mitigate common LLM pitfalls, such as hallucinations, and ensuring the explanations are both actionable and contextually relevant for non-expert users.

Figures (6)

• Figure 1: Screenshot of our chatbot application illustrating the UI components described in \ref{['sec_ui_components']}: (1) Patient Information (2) Risk Status (3) Visual Explanations (4) Chatbot Assistant (4.a) Ice-breaker Questions (4.b) Agent-Augmented Counterfactuals
• Figure 2: Plots showing the difference in objective scores for perceived actionability between novice and informed users.
• Figure 3: Difference in subjective scores for perceived actionability between novice users and informed users.
• Figure 4: This figure lists different utterance types for novice and informed users obtained using thematic analysis, along with example queries for as entered by the users.
• Figure 5: Difference in perceived understandability of augmented counterfactuals between novice and informed users.