SycEval: Evaluating LLM Sycophancy

TL;DR

The paper analyzes sycophancy in LLMs, focusing on whether models prioritize user agreement over truth in math and medical-advice tasks. It introduces a two-step evaluation framework with LLM-as-a-Judge and rebuttal-based probing to distinguish progressive versus regressive sycophancy and to measure persistence across turns. Across ChatGPT-4o, Claude-Sonnet, and Gemini-1.5-Pro on AMPS Math and MedQuad datasets, sycophancy occurred in 58.19% of cases, with preemptive rebuttals generally increasing that rate and simple rebuttals boosting progressive alignment while citation-based rebuttals amplified regressive tendencies. The findings underscore the need for targeted prompt design and model-optimization strategies to enhance correctness while mitigating harmful surface-level agreement in high-stakes settings.

Abstract

Large language models (LLMs) are increasingly applied in educational, clinical, and professional settings, but their tendency for sycophancy -- prioritizing user agreement over independent reasoning -- poses risks to reliability. This study introduces a framework to evaluate sycophantic behavior in ChatGPT-4o, Claude-Sonnet, and Gemini-1.5-Pro across AMPS (mathematics) and MedQuad (medical advice) datasets. Sycophantic behavior was observed in 58.19% of cases, with Gemini exhibiting the highest rate (62.47%) and ChatGPT the lowest (56.71%). Progressive sycophancy, leading to correct answers, occurred in 43.52% of cases, while regressive sycophancy, leading to incorrect answers, was observed in 14.66%. Preemptive rebuttals demonstrated significantly higher sycophancy rates than in-context rebuttals (61.75% vs. 56.52%, $Z=5.87$, $p<0.001$), particularly in computational tasks, where regressive sycophancy increased significantly (preemptive: 8.13%, in-context: 3.54%, $p<0.001$). Simple rebuttals maximized progressive sycophancy ($Z=6.59$, $p<0.001$), while citation-based rebuttals exhibited the highest regressive rates ($Z=6.59$, $p<0.001$). Sycophantic behavior showed high persistence (78.5%, 95% CI: [77.2%, 79.8%]) regardless of context or model. These findings emphasize the risks and opportunities of deploying LLMs in structured and dynamic domains, offering insights into prompt programming and model optimization for safer AI applications.

Figures (6)

• Figure 1: LLM-as-a-Judge Accuracy Distribution. Beta distributions modeling the expected accuracy of the LLM-as-a-Judge for both the AMPS math and MedQuad dataset
• Figure 2: Example Regressive Sycophantic Behavior. Incontext and preemptive rebuttals leading to incorrect outcomes. Green = correct; red = incorrect.
• Figure 3: Prompting Strategy: In-Context Rebuttal Formula. Structure of rebuttals presented within the same interaction window.
• Figure 4: Prompting Strategy: Preemptive Rebuttal Formula. Rebuttals anticipated and presented independently of model’s prior output.
• Figure 5: Rebuttal Generation Flow Chart. Following the initial inquiry and correct response output, LLMs were presented with additional prompts to generate rebuttals of increasing rhetorical strength. We then evaluated their responses in preemptive and in-context settings to evaluate sycophancy in language models.