Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning

Halimat Afolabi, Zainab Afolabi, Elizabeth Friel, Jude Roberts, Antonio Ji-Xu, Lloyd Chen, Egheosa Ogbomo, Emiliomo Imevbore, Phil Eneje, Wissal El Ouahidi, Aaron Sohal, Alisa Kennan, Shreya Srivastava, Anirudh Vairavan, Laura Napitu, Katie McClure

Abstract

Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment.

Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning

Abstract

Closed-source large language models (LLMs), such as ChatGPT and Gemini, are increasingly consulted for medical advice, yet their explanations may appear plausible while failing to reflect the model's underlying reasoning process. This gap poses serious risks as patients and clinicians may trust coherent but misleading explanations. We conduct a systematic black-box evaluation of faithfulness in medical reasoning among three widely used closed-source LLMs. Our study consists of three perturbation-based probes: (1) causal ablation, testing whether stated chain-of-thought (CoT) reasoning causally influences predictions; (2) positional bias, examining whether models create post-hoc justifications for answers driven by input positioning; and (3) hint injection, testing susceptibility to external suggestions. We complement these quantitative probes with a small-scale human evaluation of model responses to patient-style medical queries to examine concordance between physician assessments of explanation faithfulness and layperson perceptions of trustworthiness. We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment. In contrast, positional biases showed minimal impact in this setting. These results underscore that faithfulness, not just accuracy, must be central in evaluating LLMs for medicine, to ensure both public protection and safe clinical deployment.
Paper Structure (42 sections, 4 equations, 10 figures, 15 tables)

This paper contains 42 sections, 4 equations, 10 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Safety and Explainability of LLMs in Medicine
  4. LLM Faithfulness
  5. Methods
  6. Models
  7. Datasets
  8. Experiment 1: Causal Ablation
  9. Experiment 2: Positional Bias
  10. Experiment 3: Hint Injection
  11. Experiment 4: Human Evaluation of LLM Responses to Patient-Style Queries
  12. Statistical Analysis
  13. Results
  14. Experiment 1: Causal Ablation
  15. Experiment 2: Positional Bias
  16. ...and 27 more sections

Figures (10)

  • Figure 1: Experiments 1 (a) and 2 (b). In (b), for the unbiased and bias→gold conditions, values indicate the B-selection rate when B is the ground-truth answer. For the bias→wrong condition, values indicate the B-selection rate when B is incorrect.
  • Figure 2: Experiment 3 - accuracy (a) and hint adherence (b). In (b), stacked bars show the share of items that switched to the hinted option (green), switched away (red), or did not change (grey), for correct and incorrect hints. “No change” includes cases where the answer already matched the hint.
  • Figure 3: Experiment 4 - clinician–lay evaluations of model responses.
  • Figure 4: Physician survey (Experiment 4). Google Forms page displaying definitions and illustrative examples.
  • Figure 5: Lay participant survey (Experiment 4). Google Forms page displaying definitions and illustrative examples for each rating dimension to standardise judgments.
  • ...and 5 more figures