From Retrieving Information to Reasoning with AI: Exploring Different Interaction Modalities to Support Human-AI Coordination in Clinical Decision-Making

TL;DR

This work tackles the problem of how clinicians coordinate with large language models (LLMs) across interaction modalities in high-stakes clinical decision-making. Using a qualitative, two-phase think-aloud study with $n=12$ clinicians and real-patient vignettes, it compares free-text LLM chat, visual UI representations, and voice-based approaches. The findings show LLMs are mainly used for information retrieval rather than deliberative collaboration, but engagement increases when the model is positioned as a specialist and when AI outputs are externalized as visual reasoning artifacts; probabilistic language is generally rejected in favor of clear, actionable recommendations. Design implications emphasize non-linear information layouts, persistent information spaces, and coordinated multimodal interfaces to support attention management, rapid comparison, and collaborative reasoning in clinical workflows.

Abstract

LLMs are popular among clinicians for decision-support because of simple text-based interaction. However, their impact on clinicians' performance is ambiguous. Not knowing how clinicians use this new technology and how they compare it to traditional clinical decision-support systems (CDSS) restricts designing novel mechanisms that overcome existing tool limitations and enhance performance and experience. This qualitative study examines how clinicians (n=12) perceive different interaction modalities (text-based conversation with LLMs, interactive and static UI, and voice) for decision-support. In open-ended use of LLM-based tools, our participants took a tool-centric approach using them for information retrieval and confirmation with simple prompts instead of use as active deliberation partners that can handle complex questions. Critical engagement emerged with changes to the interaction setup. Engagement also differed with individual cognitive styles. Lastly, benefits and drawbacks of interaction with text, voice and traditional UIs for clinical decision-support show the lack of a one-size-fits-all interaction modality.

Figures (20)

• Figure 1: Overview of the patient case, clinical vignette and clinical task construction process. Real patient data serve as the starting point, from which experts select cases followed by specific clinical encounters for those vignettes based on pre-defined inclusion criteria. AI is then used to abstract and reframe selected patient vignette and corresponding encounters under controlled constraints, supporting summarization and limited variation without introducing new clinical content. Final decision-making tasks are constructed and validated by experts to preserve clinical realism.
• Figure 2: Session transcripts were segmented by task encounter and analyzed using a structured coding schema. For each coding question, five independent AI agents were prompted in parallel using the same transcript segment and schema definitions. Each agent produced a classification along with verbatim evidence quotes. These independent outputs were then reviewed through a manual check, where the researcher reconciled disagreements and determined the final coding decision used in analysis.
• Figure 3: UI Variation: Scores. Interface highlighting numerical scores and discrete decision factors. The dashboard allows clinicians to toggle between decision strategies, such as "Safety first" or "Coverage first," while presenting AI-generated match scores and color-coded rationale for different antibiotic options.
• Figure 4: UI variation: Graph. This format externalizes clinical logic by visualizing the "pathway" of a decision, showing why certain options were ruled out or remained as viable candidates based on specific clinical checkpoints like lung penetration and renal safety.
• Figure 5: UI variation: Cards. This design organizes clinical data points—such as renal safety, lung penetration, and IDSA status into side-by-side cards to support rapid comparative reasoning and identifying a leading clinical option.