Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Internal Reasoning vs. External Control: A Thermodynamic Analysis of Sycophancy in Large Language Models

Edward Y. Chang

TL;DR

This work reframes AI safety for large language models from outcome-focused alignment to inference-time process verification. It introduces Regulated Causal Anchoring (RCA), a closed-loop controller that uses an external judge to verify that model outputs causally derive from internal reasoning traces, without needing ground-truth answers. The study reveals Inverse Scaling, where stronger models show higher sycophancy on hard problems, and the Final Output Gap, where correct reasoning does not guarantee correct outputs. RCA, via Trace-Based Verification and PID-like escalation, eliminates sycophancy while maintaining or improving accuracy and can adapt to distribution shifts, offering a safety guarantee beyond traditional prompting or RLHF approaches. The work also discusses regime dynamics, safety trade-offs, and avenues for integrating inference-time control with training, aiming for higher-assurance AI in safety-critical domains.

Abstract

Large Language Models exhibit sycophancy: prioritizing agreeableness over correctness. Current remedies evaluate reasoning outcomes: RLHF rewards correct answers, self-correction critiques outputs. All require ground truth, which is often unavailable at inference time and vulnerable to the same biases. We explore evaluating the reasoning process instead. Regulated Causal Anchoring (RCA) verifies whether outputs follow from their reasoning traces, without requiring ground truth. Sycophancy manifests as trace-output inconsistency: models derive one answer but output another to please users. RCA detects this inconsistency, achieving 0.0% sycophancy while accepting 88% of valid hints. We identify two failures invisible to outcome evaluation: Inverse Scaling (frontier models sycophant more because rationalization requires capability) and the Final Output Gap (correct reasoning precedes sycophantic output). Traditional self-correction reduces these failures to 7-9% but cannot eliminate them because the model critiques itself with the same biases. RCA's process evaluation operates at inference time, requires no ground truth, and uses an independent judge that breaks the self-reinforcing bias loop: three properties that outcome evaluation lacks.

Internal Reasoning vs. External Control: A Thermodynamic Analysis of Sycophancy in Large Language Models

TL;DR

This work reframes AI safety for large language models from outcome-focused alignment to inference-time process verification. It introduces Regulated Causal Anchoring (RCA), a closed-loop controller that uses an external judge to verify that model outputs causally derive from internal reasoning traces, without needing ground-truth answers. The study reveals Inverse Scaling, where stronger models show higher sycophancy on hard problems, and the Final Output Gap, where correct reasoning does not guarantee correct outputs. RCA, via Trace-Based Verification and PID-like escalation, eliminates sycophancy while maintaining or improving accuracy and can adapt to distribution shifts, offering a safety guarantee beyond traditional prompting or RLHF approaches. The work also discusses regime dynamics, safety trade-offs, and avenues for integrating inference-time control with training, aiming for higher-assurance AI in safety-critical domains.

Abstract

Large Language Models exhibit sycophancy: prioritizing agreeableness over correctness. Current remedies evaluate reasoning outcomes: RLHF rewards correct answers, self-correction critiques outputs. All require ground truth, which is often unavailable at inference time and vulnerable to the same biases. We explore evaluating the reasoning process instead. Regulated Causal Anchoring (RCA) verifies whether outputs follow from their reasoning traces, without requiring ground truth. Sycophancy manifests as trace-output inconsistency: models derive one answer but output another to please users. RCA detects this inconsistency, achieving 0.0% sycophancy while accepting 88% of valid hints. We identify two failures invisible to outcome evaluation: Inverse Scaling (frontier models sycophant more because rationalization requires capability) and the Final Output Gap (correct reasoning precedes sycophantic output). Traditional self-correction reduces these failures to 7-9% but cannot eliminate them because the model critiques itself with the same biases. RCA's process evaluation operates at inference time, requires no ground truth, and uses an independent judge that breaks the self-reinforcing bias loop: three properties that outcome evaluation lacks.
Paper Structure (134 sections, 3 equations, 1 figure, 15 tables, 1 algorithm)

This paper contains 134 sections, 3 equations, 1 figure, 15 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The Limitation of Outcome Evaluation.
  3. Two Empirical Failures.
  4. From Outcome to Process Evaluation.
  5. Our Approach: RCA.
  6. Validation.
  7. Contributions.
  8. Regime Classification.
  9. Related Work
  10. Observational Prompting.
  11. Unfaithful Traces and Sycophancy.
  12. Iterative Self-Correction.
  13. Outcome-Based Training.
  14. Multi-Agent Verification.
  15. Control-Theoretic Perspectives.
  16. ...and 119 more sections

Figures (1)

  • Figure 1: RCA Control Flow. On failure, the PID controller shifts persona, escalates strategy, and retries. Safety Fallback ensures no sycophantic output escapes.