MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication

TL;DR

MedRedFlag introduces a large-scale, real-world dataset of patient questions containing implicit false premises and clinician-led redirection. It presents a semi-automatic pipeline to surface redirection using GPT-5, and benchmarks multiple LLMs against clinician redirection, revealing a substantial safety gap: even when models detect false premises, they often still provide unsafe, premise-reinforcing guidance. The work also evaluates mitigation strategies, showing that identification alone is insufficient and that even oracle-based redirection does not fully prevent harmful accommodation. These findings underscore the need for improved alignment and interaction strategies to prioritize patient safety in AI-assisted medical advice, with MedRedFlag and accompanying tools serving as a benchmark for future safety improvements.

Abstract

Real-world health questions from patients often unintentionally embed false assumptions or premises. In such cases, safe medical communication typically involves redirection: addressing the implicit misconception and then responding to the underlying patient context, rather than the original question. While large language models (LLMs) are increasingly being used by lay users for medical advice, they have not yet been tested for this crucial competency. Therefore, in this work, we investigate how LLMs react to false premises embedded within real-world health questions. We develop a semi-automated pipeline to curate MedRedFlag, a dataset of 1100+ questions sourced from Reddit that require redirection. We then systematically compare responses from state-of-the-art LLMs to those from clinicians. Our analysis reveals that LLMs often fail to redirect problematic questions, even when the problematic premise is detected, and provide answers that could lead to suboptimal medical decision making. Our benchmark and results reveal a novel and substantial gap in how LLMs perform under the conditions of real-world health communication, highlighting critical safety concerns for patient-facing medical AI systems. Code and dataset are available at https://github.com/srsambara-1/MedRedFlag.

Figures (3)

• Figure 1: MedRedFlag contains patient questions with false underlying assumptions that human clinicians choose to redirect when answering. LLMs often accommodate false assumptions when answering instead.
• Figure 2: (A) Automated redirection annotation pipeline for constructing MedRedFlag. Using GPT-5, the pipeline automatically annotates input QA pairs to detect redirection by identifying cases where a summarized (i) initial patient question differs substantively from the (ii) implicit question answered by the physician, then (iii, iv) summarizes key misconceptions redirected in the response. (B) Additional examples of input QA pairs and automatically summarized questions tagged for redirection. Example QA pairs are based on real instances but altered for data privacy; shown with actual pipeline outputs on these inputs.
• Figure 3: Anatomy of a representative LLM response to patient question with embedded false assumptions. We find that even when LLMs address false or unsafe assumptions in the patient question (green), they still often extensively accommodate the false assumption(red) with detailed, unsafe advice based on the patient question.