PARROT: Persuasion and Agreement Robustness Rating of Output Truth -- A Sycophancy Robustness Benchmark for LLMs

TL;DR

The paper tackles the vulnerability of LLMs to social pressure that promotes sycophancy, proposing PARROT, a dual-path, calibration-aware framework to quantify how accuracy degrades under authoritative false claims. PARROT uses deterministic manipulation, logprob-based confidence tracking, and an eight-state behavioral taxonomy to classify failure modes, enabling reproducible, cross-model comparisons across 22 models and 13 domains. The results reveal pronounced heterogeneity: frontier models exhibit strong resistance to social pressure, while smaller and older models suffer dramatic epistemic collapse, with domain-specific patterns showing higher fragility in legal, global-facts, and philosophy domains. The work argues that resilience to over-coordination pressure should be a central objective in alignment and safety pipelines, alongside traditional goals like accuracy and privacy, to ensure reliable deployment in high-stakes settings.

Abstract

This study presents PARROT (Persuasion and Agreement Robustness Rating of Output Truth), a robustness focused framework designed to measure the degradation in accuracy that occurs under social pressure exerted on users through authority and persuasion in large language models (LLMs) the phenomenon of sycophancy (excessive conformity). PARROT (i) isolates causal effects by comparing the neutral version of the same question with an authoritatively false version using a double-blind evaluation, (ii) quantifies confidence shifts toward the correct and imposed false responses using log-likelihood-based calibration tracking, and (iii) systematically classifies failure modes (e.g., robust correct, sycophantic agreement, reinforced error, stubborn error, self-correction, etc.) using an eight-state behavioral taxonomy. We evaluated 22 models using 1,302 MMLU-style multiple-choice questions across 13 domains and domain-specific authority templates. Findings show marked heterogeneity: advanced models (e.g., GPT-5, GPT-4.1, Claude Sonnet 4.5) exhibit low "follow rates" ($\leq 11\%$, GPT-5: 4\%) and minimal accuracy loss, while older/smaller models show severe epistemic collapse (GPT-4: 80\%, Qwen 2.5-1.5B: 94\%). The danger is not limited to response changes; weak models reduce confidence in the correct response while increasing confidence in the imposed incorrect response. While international law and global knowledge at the domain level exhibit high fragility, elementary mathematics is relatively resilient. Consequently, we argue that the goal of "resistance to overfitting pressure" should be addressed as a primary objective alongside accuracy, harm avoidance, and privacy for safe deployment in the real world.

Figures (2)

• Figure 1: Follow Rate vs. Baseline Accuracy, sized by Confidence Inflation on Asserted Errors.
• Figure 2: Domain-specific accuracy degradation under manipulation across 22 models and 13 academic domains.