Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Overalignment in Frontier LLMs: An Empirical Study of Sycophantic Behaviour in Healthcare

Clément Christophe, Wadood Mohammed Abdul, Prateek Munjal, Tathagata Raha, Ronnie Rajan, Praveenkumar Kanithi

TL;DR

This paper investigates sycophancy in frontier LLMs within healthcare and introduces the Adjusted Sycophancy Score $S_a$, a noise-aware metric that isolates alignment bias from model confusability using medical MCQA benchmarks with verifiable ground truths. Through scaling analyses across the Qwen-3 and Llama-3 families and experiments with vanilla and perturbation prompts, the authors reveal that higher parameter counts yield greater clinical resilience, with $S_a$ stabilizing near zero beyond certain scales. They also uncover a vulnerability in reasoning-optimized 'Thinking' models: while vanilla accuracy improves, internal reasoning traces can rationalize incorrect user suggestions under authoritative pressure, undermining robustness. The findings suggest benchmark accuracy does not guarantee clinical reliability and motivate alignment strategies that prioritize epistemic integrity over user deference.

Abstract

As LLMs are increasingly integrated into clinical workflows, their tendency for sycophancy, prioritizing user agreement over factual accuracy, poses significant risks to patient safety. While existing evaluations often rely on subjective datasets, we introduce a robust framework grounded in medical MCQA with verifiable ground truths. We propose the Adjusted Sycophancy Score, a novel metric that isolates alignment bias by accounting for stochastic model instability, or "confusability". Through an extensive scaling analysis of the Qwen-3 and Llama-3 families, we identify a clear scaling trajectory for resilience. Furthermore, we reveal a counter-intuitive vulnerability in reasoning-optimized "Thinking" models: while they demonstrate high vanilla accuracy, their internal reasoning traces frequently rationalize incorrect user suggestions under authoritative pressure. Our results across frontier models suggest that benchmark performance is not a proxy for clinical reliability, and that simplified reasoning structures may offer superior robustness against expert-driven sycophancy.

Overalignment in Frontier LLMs: An Empirical Study of Sycophantic Behaviour in Healthcare

TL;DR

This paper investigates sycophancy in frontier LLMs within healthcare and introduces the Adjusted Sycophancy Score , a noise-aware metric that isolates alignment bias from model confusability using medical MCQA benchmarks with verifiable ground truths. Through scaling analyses across the Qwen-3 and Llama-3 families and experiments with vanilla and perturbation prompts, the authors reveal that higher parameter counts yield greater clinical resilience, with stabilizing near zero beyond certain scales. They also uncover a vulnerability in reasoning-optimized 'Thinking' models: while vanilla accuracy improves, internal reasoning traces can rationalize incorrect user suggestions under authoritative pressure, undermining robustness. The findings suggest benchmark accuracy does not guarantee clinical reliability and motivate alignment strategies that prioritize epistemic integrity over user deference.

Abstract

As LLMs are increasingly integrated into clinical workflows, their tendency for sycophancy, prioritizing user agreement over factual accuracy, poses significant risks to patient safety. While existing evaluations often rely on subjective datasets, we introduce a robust framework grounded in medical MCQA with verifiable ground truths. We propose the Adjusted Sycophancy Score, a novel metric that isolates alignment bias by accounting for stochastic model instability, or "confusability". Through an extensive scaling analysis of the Qwen-3 and Llama-3 families, we identify a clear scaling trajectory for resilience. Furthermore, we reveal a counter-intuitive vulnerability in reasoning-optimized "Thinking" models: while they demonstrate high vanilla accuracy, their internal reasoning traces frequently rationalize incorrect user suggestions under authoritative pressure. Our results across frontier models suggest that benchmark performance is not a proxy for clinical reliability, and that simplified reasoning structures may offer superior robustness against expert-driven sycophancy.
Paper Structure (17 sections, 3 equations, 4 figures, 3 tables)

This paper contains 17 sections, 3 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Experimental Design
  5. Sycophancy Score
  6. Results
  7. Experimental Setup and Model Selection.
  8. Finding 1: Scaling Laws.
  9. Finding 2: The Vulnerability of Reasoning Traces.
  10. Finding 3: Benchmark Maturity and Authority Resilience.
  11. Conclusion
  12. Limitations
  13. Benchmark and Linguistic Scope.
  14. Simplification of User Authority.
  15. Assumptions in Noise Adjustment.
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Raw ($S_{r}$) and Adjusted ($S_{a}$) Sycophancy Scores across the Qwen-3 model family.
  • Figure 2: Raw ($S_{r}$) and Adjusted ($S_{a}$) Sycophancy Scores across the Llama-3 model family.
  • Figure 3: $S_a$ score and accuracy for both Instruct and Thinking Qwen-3 models on MedQA. Thinking models show superior accuracy but a fragile resilience to perceived authority.
  • Figure 4: $S_a$ score and accuracy for both Instruct and Thinking Qwen-3 models on MedQA when the role is in the System Prompt. Thinking models show no particular behavior change compared to the basic nudge.