Towards Better Health Conversations: The Benefits of Context-seeking

TL;DR

The paper investigates how proactive context-seeking in conversational AIs affects health information seeking by laypeople. Through four mixed-methods studies (three qualitative, one quantitative) and a deliberately designed Wayfinding AI, it demonstrates that eliciting user context during dialogue leads to more relevant, tailored, and helpful health information, albeit with longer conversations. The Wayfinding AI consistently outperformed a Baseline AI on perceived usefulness and engagement, highlighting concrete design patterns for context- gathering prompts and UI presentation. These findings illuminate actionable design considerations for health chatbot UX and dialogue management, with implications for improving access to high-quality health information while acknowledging risks and the need for broader evaluation of health outcomes.

Abstract

Navigating health questions can be daunting in the modern information landscape. Large language models (LLMs) may provide tailored, accessible information, but also risk being inaccurate, biased or misleading. We present insights from 4 mixed-methods studies (total N=163), examining how people interact with LLMs for their own health questions. Qualitative studies revealed the importance of context-seeking in conversational AIs to elicit specific details a person may not volunteer or know to share. Context-seeking by LLMs was valued by participants, even if it meant deferring an answer for several turns. Incorporating these insights, we developed a "Wayfinding AI" to proactively solicit context. In a randomized, blinded study, participants rated the Wayfinding AI as more helpful, relevant, and tailored to their concerns compared to a baseline AI. These results demonstrate the strong impact of proactive context-seeking on conversational dynamics, and suggest design patterns for conversational AI to help navigate health topics.

Figures (8)

• Figure 1: Screenshots of the two AI chatbot designs used in Studies 3 and 4. Top shows a baseline, 1-column AI design. Bottom shows a 2-column design in which the current answer is shown in the right column, while clarifying questions are displayed underneath user prompts on the left.
• Figure 2: Quantitative study design (Study 4). All participants used both AIs used in the study (Wayfinding AI and Baseline AI) to better understand their own health-related questions; but were randomly assigned in the order with which to use them. After using each AI, participants answered questions about their satisfaction with the experience; at the end of the survey, subjects explicitly compared the two AIs and expressed which they would prefer along a range of axes.
• Figure 3: Summary of satisfaction measures for Wayfinding AI and Baseline AI. Values reflect the percent of respondents who report being somewhat or very satisfied with each agent. Error bars represent 95% confidence intervals determined by bootstrap. Asterisks indicate significant difference by a Wilcoxon test, at p < 0.05, with Bonferroni correction for multiple comparisons.
• Figure 4: Summary of side-by-side comparison between Wayfinding AI and Baseline AI. A, distributions of survey responses for each preference question; B, Summary of preference distributions. Values reflect the arithmetic mean of Likert scale responses on a scale of [-2, +2]; negative values indicate preference for Baseline AI, positive values indicate preference for Wayfinding AI. Error bars represent 95% confidence intervals determined by bootstrap.
• Figure 5: Sankey flow diagrams illustrating distributions of user prompt types across conversations in Study 4. Top, summary of all conversations with Baseline AI. Bottom, summary of all conversations with Wayfinding AI. Nodes indicate different user prompt types. Thickness of lines connecting two nodes indicates relative proportion of conversations with the first prompt type followed by the second prompt type. The first 5 turns are shown. The abbreviations describing each turn type are described in more detail in the Appendix. Abbreviations: ANSWER_CQ: Answering clarifying questions asked by the AI; ELABORATION: Request for AI to elaborate on an aspect of its response; REL_TOPIC: Related topic exploration; CLARIFY: user provides refinement or clarification of their need without prompting; PIVOT: User pivots to a new topic; COMPARE: user asks AI to compare two entities related to the question; NEXT_STEP: User asks about a task-related next step; VERIFY: User expresses uncertainty or seeks confirmation of information provided. Details on how labels were applied to conversations are in the Appendix.