Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics

Yinghao Zhu, Dehao Sui, Zixiang Wang, Xuning Hu, Lei Gu, Yifan Qi, Tianchen Wu, Ling Wang, Yuan Wei, Wen Tang, Zhihan Cui, Yasha Wang, Lequan Yu, Ewen M Harrison, Junyi Gao, Liantao Ma

TL;DR

This study reframes medical AI adoption from pure accuracy to sociotechnical collaboration by introducing AICare, an interactive and interpretable AI copilot for longitudinal EHR contexts. Grounded in nephrology and obstetrics, AICare combines dynamic risk trajectories, feature-level explanations, and LLM-driven narratives to support clinicians in hypothesis generation and verification rather than dictating decisions. Across a real-world, within-subject study with 16 clinicians, AICare reduced perceived cognitive workload and raised diagnostic confidence without compromising accuracy, while revealing distinct interaction strategies: novices use it as a scaffolding tool, whereas experts perform adversarial verification. The findings yield design implications for next-generation clinical copilots, emphasizing context-aware disclosure, robust grounding of generative outputs, and maintaining clinician agency to ensure safe, trusted integration into routine practice.

Abstract

Clinician skepticism toward opaque AI hinders adoption in high-stakes healthcare. We present AICare, an interactive and interpretable AI copilot for collaborative clinical decision-making. By analyzing longitudinal electronic health records, AICare grounds dynamic risk predictions in scrutable visualizations and LLM-driven diagnostic recommendations. Through a within-subjects counterbalanced study with 16 clinicians across nephrology and obstetrics, we comprehensively evaluated AICare using objective measures (task completion time and error rate), subjective assessments (NASA-TLX, SUS, and confidence ratings), and semi-structured interviews. Our findings indicate AICare's reduced cognitive workload. Beyond performance metrics, qualitative analysis reveals that trust is actively constructed through verification, with interaction strategies diverging by expertise: junior clinicians used the system as cognitive scaffolding to structure their analysis, while experts engaged in adversarial verification to challenge the AI's logic. This work offers design implications for creating AI systems that function as transparent partners, accommodating diverse reasoning styles to augment rather than replace clinical judgment.

Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics

TL;DR

This study reframes medical AI adoption from pure accuracy to sociotechnical collaboration by introducing AICare, an interactive and interpretable AI copilot for longitudinal EHR contexts. Grounded in nephrology and obstetrics, AICare combines dynamic risk trajectories, feature-level explanations, and LLM-driven narratives to support clinicians in hypothesis generation and verification rather than dictating decisions. Across a real-world, within-subject study with 16 clinicians, AICare reduced perceived cognitive workload and raised diagnostic confidence without compromising accuracy, while revealing distinct interaction strategies: novices use it as a scaffolding tool, whereas experts perform adversarial verification. The findings yield design implications for next-generation clinical copilots, emphasizing context-aware disclosure, robust grounding of generative outputs, and maintaining clinician agency to ensure safe, trusted integration into routine practice.

Abstract

Clinician skepticism toward opaque AI hinders adoption in high-stakes healthcare. We present AICare, an interactive and interpretable AI copilot for collaborative clinical decision-making. By analyzing longitudinal electronic health records, AICare grounds dynamic risk predictions in scrutable visualizations and LLM-driven diagnostic recommendations. Through a within-subjects counterbalanced study with 16 clinicians across nephrology and obstetrics, we comprehensively evaluated AICare using objective measures (task completion time and error rate), subjective assessments (NASA-TLX, SUS, and confidence ratings), and semi-structured interviews. Our findings indicate AICare's reduced cognitive workload. Beyond performance metrics, qualitative analysis reveals that trust is actively constructed through verification, with interaction strategies diverging by expertise: junior clinicians used the system as cognitive scaffolding to structure their analysis, while experts engaged in adversarial verification to challenge the AI's logic. This work offers design implications for creating AI systems that function as transparent partners, accommodating diverse reasoning styles to augment rather than replace clinical judgment.
Paper Structure (67 sections, 6 figures, 18 tables)

This paper contains 67 sections, 6 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human-AI Collaboration in Clinical Workflows
  4. Interpretability and Trust Calibration in Medical AI
  5. Impact of Clinical Context and Expertise on AI Interaction
  6. The AICare System
  7. Underlying Interpretable Predictive EHR Model
  8. System Features
  9. Dynamic risk trajectory visualization.
  10. Interactive list of critical risk factors.
  11. LLM-driven diagnostic recommendation.
  12. Population-level indicator analysis.
  13. Implementation Details
  14. Real-world clinical datasets and predictive tasks.
  15. Validation of predictive performance benchmarks.
  16. ...and 52 more sections

Figures (6)

  • Figure 1: The AICare system interface, designed as an interactive and interpretable AI copilot. The dashboard features (A) a dynamic risk trajectory visualization over the patient's visits that externalizes memory-intensive trend analysis, (B) an interactive list of critical risk factors with their adaptive importance scores enabling immediate drill-down into evidence, (C) an LLM-driven diagnostic recommendation synthesizing key findings, and (D) a population-level indicator analysis providing cohort context for a selected feature. Interaction patterns reveal that these features support dual strategies: acting as cognitive scaffolding for novices to structure analysis, while enabling experts to perform "adversarial verification" of the model's logic.
  • Figure 2: An overview of the user study methodology, from data collection to evaluation. We utilized structured EHR data from three institutions across two specialties (Nephrology and Obstetrics) to train the AICare system for two predictive tasks. The study involved 16 clinicians in a rigorous within-subjects, counterbalanced design. Participants completed risk assessment tasks under two conditions: AI-assisted (using AICare) and unassisted (a baseline). We collected quantitative and qualitative data through post-task questionnaires and semi-structured interviews to evaluate AICare's impact on efficiency, accuracy, cognitive workload, trust, and perceived clinical value. Quantitative results demonstrate that AICare significantly reduces cognitive workload by offloading the mental burden of synthesizing longitudinal data, without compromising diagnostic accuracy.
  • Figure 3: Thematic map of clinician interview findings. The hierarchical structure organizes key themes and sub-themes derived from semi-structured interviews. Findings are grouped by each of the three core research questions (RQs) and are supported by representative quotes from participants. Qualitative analysis confirms that transparency amplifies trust when the reasoning is clinically plausible, but accelerates rejection when the AI contradicts established medical logic.
  • Figure 4: Study framework mapping the progression from research questions to results and discussion. The diagram illustrates three core pathways: the shift from prediction to explanatory power (RQ1), the divergence of interaction strategies based on clinical expertise (RQ2), and the active construction of trust through verification (RQ3), culminating in implications for cognitive processing and system design.
  • Figure 5: The flowchart of participant selection for nephrology cohorts. The left panel displays the screening process for XY Hospital, while the right panel depicts the process for BS Hospital. The procedure sequentially filters patients based on age, dialysis vintage, history of other renal therapies, and presence of malignant tumors.
  • ...and 1 more figures