Engaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-Making

TL;DR

This study investigates how interface design shapes human-AI collaboration in high-stakes diabetes decision-making. It compares six decision-support mechanisms, spanning textual and visual explanations (XAI) and cognitive forcing functions (CFFs), to assess effects on engagement, trust calibration, and decision accuracy. Key findings show that mechanisms like AI confidence levels and performance visualizations can improve performance and engagement, while more reflective approaches (AI-driven questions, human feedback) increase cognitive load and can reduce trust or accuracy. The results highlight a trade-off between transparency, cognitive effort, and trust, underscoring the need for adaptive interfaces that balance System 1 intuition with System 2 deliberation to optimize human-AI collaboration in safety-critical tasks.

Abstract

As reliance on AI systems for decision-making grows, it becomes critical to ensure that human users can appropriately balance trust in AI suggestions with their own judgment, especially in high-stakes domains like healthcare. However, human + AI teams have been shown to perform worse than AI alone, with evidence indicating automation bias as the reason for poorer performance, particularly because humans tend to follow AI's recommendations even when they are incorrect. In many existing human + AI systems, decision-making support is typically provided in the form of text explanations (XAI) to help users understand the AI's reasoning. Since human decision-making often relies on System 1 thinking, users may ignore or insufficiently engage with the explanations, leading to poor decision-making. Previous research suggests that there is a need for new approaches that encourage users to engage with the explanations and one proposed method is the use of cognitive forcing functions (CFFs). In this work, we examine how various decision-support mechanisms impact user engagement, trust, and human-AI collaborative task performance in a diabetes management decision-making scenario. In a controlled experiment with 108 participants, we evaluated the effects of six decision-support mechanisms split into two categories of explanations (text, visual) and four CFFs. Our findings reveal that mechanisms like AI confidence levels, text explanations, and performance visualizations enhanced human-AI collaborative task performance, and improved trust when AI reasoning clues were provided. Mechanisms like human feedback and AI-driven questions encouraged deeper reflection but often reduced task performance by increasing cognitive effort, which in turn affected trust. Simple mechanisms like visual explanations had little effect on trust, highlighting the importance of striking a balance in CFF and XAI design.

Figures (7)

• Figure 1: Dual-layer framework guiding the transition from System 1 (intuitive thinking) to System 2 (deliberative thinking) with decision support mechanisms. We study six decision support mechanisms, aiming to explore how explanations of varying cognitive load influence user engagement. These mechanisms are designed to encourage a transition from System 1 to System 2 processing. These mechanisms include: C1 - Textual Explanation, C2 - Visual Explanation, C3 - AI Confidence Levels (CLs), C4 - Human Feedback, C5 - AI-Driven Questions, and C6 - Performance Visualization. The framework categorizes mechanisms under XAI and CFFs.
• Figure 1: McNemar Test Results for User Decisions with AI Suggestions.
• Figure 2: Visualization of User Answer Correctness.
• Figure 3: Questionnaire scores across six conditions on the five measures: satisfaction, system complexity, reliability, trust, and system accuracy. Error bars represent variability, and horizontal bars indicate statistical significance ($p<0.05$) where applicable.
• Figure 4: Interface design across different phases: Phase 1, where the user makes an independent decision without AI assistance; Phase 2, where the AI presents its suggestion but without any explanation; and Phase 3, where condition-specific AI decision-making support mechanisms are applied, followed by another chance for users to update their decision. The right side of the figure illustrates the various explanation methods employed. (1) Textual explanation for why the AI selected a preferred meal, (2) Visual segmentation and labeling of meal items, (3) AI's confidence level (CL) and an estimation the user's confidence level (CL), (4) Human feedback on the AI and their own confidence levels (CLs), (5) AI prompted questions for the user, and (6) Performance visualization of the correctness of past questions, for both the AI and the user.