Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Health Professionals' Onboarding with AI and XAI for Trustworthy Human-AI Collaborative Decision Making

Min Hun Lee, Silvana Xin Yi Choo, Shamala D/O Thilarajah

TL;DR

This paper investigates how to onboard health professionals and students with AI and explainable AI (XAI) in high-stakes stroke rehabilitation. It develops onboarding tutorial materials that explain AI inputs/outputs, data, performance, and three local XAI explanations (feature importance, counterfactuals, prototypes) and tests them via semi-structured interviews with 16 participants. Key findings show a demand for benchmark information, practical AI benefits, and iterative interaction trials to calibrate trust, with explanations varying in usefulness between onboarding and decision support. The work contributes design recommendations, emphasizes user-centered explanations and governance (audits, multi-site validation), and highlights the need for metrics of user understanding beyond traditional performance measures to enable trustworthy human-AI collaboration in healthcare.

Abstract

With advanced AI/ML, there has been growing research on explainable AI (XAI) and studies on how humans interact with AI and XAI for effective human-AI collaborative decision-making. However, we still have a lack of understanding of how AI systems and XAI should be first presented to users without technical backgrounds. In this paper, we present the findings of semi-structured interviews with health professionals (n=12) and students (n=4) majoring in medicine and health to study how to improve onboarding with AI and XAI. For the interviews, we built upon human-AI interaction guidelines to create onboarding materials of an AI system for stroke rehabilitation assessment and AI explanations and introduce them to the participants. Our findings reveal that beyond presenting traditional performance metrics on AI, participants desired benchmark information, the practical benefits of AI, and interaction trials to better contextualize AI performance, and refine the objectives and performance of AI. Based on these findings, we highlight directions for improving onboarding with AI and XAI and human-AI collaborative decision-making.

Improving Health Professionals' Onboarding with AI and XAI for Trustworthy Human-AI Collaborative Decision Making

TL;DR

This paper investigates how to onboard health professionals and students with AI and explainable AI (XAI) in high-stakes stroke rehabilitation. It develops onboarding tutorial materials that explain AI inputs/outputs, data, performance, and three local XAI explanations (feature importance, counterfactuals, prototypes) and tests them via semi-structured interviews with 16 participants. Key findings show a demand for benchmark information, practical AI benefits, and iterative interaction trials to calibrate trust, with explanations varying in usefulness between onboarding and decision support. The work contributes design recommendations, emphasizes user-centered explanations and governance (audits, multi-site validation), and highlights the need for metrics of user understanding beyond traditional performance measures to enable trustworthy human-AI collaboration in healthcare.

Abstract

With advanced AI/ML, there has been growing research on explainable AI (XAI) and studies on how humans interact with AI and XAI for effective human-AI collaborative decision-making. However, we still have a lack of understanding of how AI systems and XAI should be first presented to users without technical backgrounds. In this paper, we present the findings of semi-structured interviews with health professionals (n=12) and students (n=4) majoring in medicine and health to study how to improve onboarding with AI and XAI. For the interviews, we built upon human-AI interaction guidelines to create onboarding materials of an AI system for stroke rehabilitation assessment and AI explanations and introduce them to the participants. Our findings reveal that beyond presenting traditional performance metrics on AI, participants desired benchmark information, the practical benefits of AI, and interaction trials to better contextualize AI performance, and refine the objectives and performance of AI. Based on these findings, we highlight directions for improving onboarding with AI and XAI and human-AI collaborative decision-making.
Paper Structure (42 sections, 5 figures, 4 tables)

This paper contains 42 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Challenges of Deploying AI-based Decision Support Systems for Human-AI Collaboration
  4. Towards Human-Centered, Trustworthy, and Explainable AI
  5. Designing Human-Centered AI
  6. Efforts on Framework for Trustworthy AI
  7. Technically Oriented AI Explanations
  8. Study Design
  9. Onboarding Tutorial Materials
  10. Recruitment and Demographics
  11. Protocol
  12. Data Analysis
  13. Results
  14. Clinical Practices of Therapy & Trustworthy Relationships
  15. Clinical Practices
  16. ...and 27 more sections

Figures (5)

  • Figure 1: Onboarding Tutorial Materials of an AI that introduce (a) an (c) the context of physical stroke rehabilitation assessment and (b) AI applications of physical stroke rehabilitation assessment and therapy.
  • Figure 2: Onboarding Tutorial Materials of an AI: (a) a diagram of how AI is developed and operated, (b) dataset, and (c) evaluation metrics and performance
  • Figure 3: Onboarding Tutorial Materials of an AI explanations: (a) motivation of an AI explanation, (b) a feature importance explanation, (c) a counterfactual explanation, and (d) prototype/example-based explanations.
  • Figure 4: Characteristics and skills of a trustworthy colleague and a process to build a trustworthy relationship with colleagues: a trustworthy relationship requires soft and hard skills to have interactive communications over time to understand colleagues' roles, skills, and goals and add a value on their goals.
  • Figure 5: Ratio of Rankings on the Perceived Usefulness of AI Explanations (Feature Importance, Counterfactuals, and Example-based) for Onboarding and Decision Support with AI from all participants, therapists with experience of stroke rehabilitation, and others (other health professionals and students majoring in medicine and health).