Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

TL;DR

This work investigates how LLMs reason when exposed to counterfactual medical evidence by introducing MedCounterFact, a dataset that perturbs real RCT evidence with four counterfactual categories. Across nine frontier models and multiple prompt styles, models largely accept counterfactual evidence and produce confident, potentially unsafe conclusions, with limited recognition of implausibility or safety warnings. The results reveal that scaling and medical fine-tuning do not reliably enhance safety-aware, evidence-grounded reasoning, suggesting there is currently no clear boundary between faithfulness to context and safety in medical reasoning. The study highlights a critical vulnerability in evidence-grounded LLMs and calls for robust safety fallbacks and methodological safeguards in high-stakes domains.

Abstract

In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the context does not align with model priors or safety protocols? In this paper, we investigate how LLMs behave and reason when presented with counterfactual or even adversarial medical evidence. We first construct MedCounterFact, a counterfactual medical QA dataset that requires the models to answer clinical comparison questions (i.e., judge the efficacy of certain treatments, with evidence consisting of randomized controlled trials provided as context). In MedCounterFact, real-world medical interventions within the questions and evidence are systematically replaced with four types of counterfactual stimuli, ranging from unknown words to toxic substances. Our evaluation across multiple frontier LLMs on MedCounterFact reveals that in the presence of counterfactual evidence, existing models overwhelmingly accept such "evidence" at face value even when it is dangerous or implausible, and provide confident and uncaveated answers. While it may be prudent to draw a boundary between faithfulness and safety, our findings reveal that there exists no such boundary yet.

Figures (22)

• Figure 1: In evidence-based medical QA, models need to synthesize evidence (often RCTs) to provide an answer. This paper explores the influence of counterfactual evidence, which we found to override prior safety constraints in LLMs. Evidence Adherence rate (EA rate) measures how strongly a model adheres to the provided evidence.
• Figure 2: Overview of the four counterfactual intervention categories ( NONCE , MEDICAL , NON-MEDICAL , and TOXIC ). Each instance contains a valid intervention ($T$) in a clinical question; the model needs to reason through the evidence (RCTs) to arrive at an answer label (Higher, Lower, No Difference, or Uncertain). We replace $T$ with counterfactual terms ($T’$) to obtain the counterfactual question ($Q'$), and evaluate model responses. Note: exemestane is a type of aromatase inhibitor.
• Figure 3: Box plots showing the Uncertain rate and the Evidence Adherence rate (EA rate) across multiple-choice and free-form response formats, aggregated over all models for each original intervention and counterfactual ones. Prompt variants: No-Evd, Evd, Expert+Evd and Skept+Evd. Introducing evidence systematically lowers Uncertain rates and increases EA rates, even with adversarial counterfactual evidence that violates safety constraints or common sense.
• Figure 4: Mean $\Delta$Uncertain rates versus model sizes across counterfactual categories using Skept+Evd prompt (multiple-choice setting). Model sizes for Gemini-2.5-Flash and GPT-5-mini are estimated.
• Figure 5: Distribution of predicted probabilities for each answer class across different types of counterfactual interventions for OLMo-3-7B-Instruct. (a) Without evidence in context, probabilities are distributed broadly across answer classes, with high variance within each perturbation category. (b) When evidence is provided in context, distributions shrink and shift similarly across perturbations and evidence variants.