Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

T3: Benchmarking Sycophancy and Skepticism in Causal Judgment

Edward Y. Chang

TL;DR

T3 (Testing Trustworthy Thinking) introduces a diagnostic benchmark to evaluate LLM causal judgment across Pearl's Ladder of Causality, decomposing performance into Utility and Safety and incorporating Wise Refusal on AMBIGUOUS cases. It uses 454 expert-curated vignettes to reveal distinct failure modes, including the Skepticism Trap and a Scaling Paradox on counterfactuals, and demonstrates that a process-verified protocol (RCA) can restore decisive causal judgment. The framework combines three prompting protocols to separate capability from robustness and provides a rigorous Sheep/Wolf analysis to expose safety-driven refusals and endorsements. The findings have implications for alignment and benchmarking, showing that safety tuning can trade off genuine causal reasoning and that structured verification can mitigate these pathologies. Overall, T3 offers a granular, diagnostic tool for improving trustworthy causal reasoning in LLMs and guiding future alignment research.

Abstract

We introduce T3 (Testing Trustworthy Thinking), a diagnostic benchmark designed to rigorously evaluate LLM causal judgment across Pearl's Ladder of Causality. Comprising 454 expert-curated vignettes, T3 prioritizes high-resolution failure analysis, decomposing performance into Utility (sensitivity), Safety (specificity), and Wise Refusal on underdetermined cases. By applying T3 to frontier models, we diagnose two distinct pathologies: a "Skepticism Trap" at L1 (where safety-tuned models like Claude Haiku reject 60% of valid links) and a non-monotonic Scaling Paradox at L3. In the latter, the larger GPT-5.2 underperforms GPT-4-Turbo by 55 points on ambiguous counterfactuals, driven by a collapse into paralysis (excessive hedging) rather than hallucination. Finally, we use the benchmark to validate a process-verified protocol (RCA), showing that T3 successfully captures the restoration of decisive causal judgment under structured verification.

T3: Benchmarking Sycophancy and Skepticism in Causal Judgment

TL;DR

T3 (Testing Trustworthy Thinking) introduces a diagnostic benchmark to evaluate LLM causal judgment across Pearl's Ladder of Causality, decomposing performance into Utility and Safety and incorporating Wise Refusal on AMBIGUOUS cases. It uses 454 expert-curated vignettes to reveal distinct failure modes, including the Skepticism Trap and a Scaling Paradox on counterfactuals, and demonstrates that a process-verified protocol (RCA) can restore decisive causal judgment. The framework combines three prompting protocols to separate capability from robustness and provides a rigorous Sheep/Wolf analysis to expose safety-driven refusals and endorsements. The findings have implications for alignment and benchmarking, showing that safety tuning can trade off genuine causal reasoning and that structured verification can mitigate these pathologies. Overall, T3 offers a granular, diagnostic tool for improving trustworthy causal reasoning in LLMs and guiding future alignment research.

Abstract

We introduce T3 (Testing Trustworthy Thinking), a diagnostic benchmark designed to rigorously evaluate LLM causal judgment across Pearl's Ladder of Causality. Comprising 454 expert-curated vignettes, T3 prioritizes high-resolution failure analysis, decomposing performance into Utility (sensitivity), Safety (specificity), and Wise Refusal on underdetermined cases. By applying T3 to frontier models, we diagnose two distinct pathologies: a "Skepticism Trap" at L1 (where safety-tuned models like Claude Haiku reject 60% of valid links) and a non-monotonic Scaling Paradox at L3. In the latter, the larger GPT-5.2 underperforms GPT-4-Turbo by 55 points on ambiguous counterfactuals, driven by a collapse into paralysis (excessive hedging) rather than hallucination. Finally, we use the benchmark to validate a process-verified protocol (RCA), showing that T3 successfully captures the restoration of decisive causal judgment under structured verification.
Paper Structure (118 sections, 1 equation, 4 figures, 11 tables, 1 algorithm)

This paper contains 118 sections, 1 equation, 4 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Relation to Prior Work
  3. Causal reasoning benchmarks.
  4. Sycophancy and truthfulness.
  5. Sensitivity vs. Specificity in Causal Judgment
  6. Key Findings
  7. Contributions
  8. Related Work
  9. Causal Reasoning Benchmarks
  10. Foundations and Formalism.
  11. Pearl's ladder benchmarks.
  12. Causal discovery and association.
  13. Commonsense causality.
  14. Sycophancy and Truthfulness
  15. The Sycophancy Problem.
  16. ...and 103 more sections

Figures (4)

  • Figure 1: Anatomy of T3 Vignettes. We test discernment by pairing valid causal links (Sheep) with structural traps (Wolves, Example 1) and underdetermined scenarios requiring calibrated refusal (Example 2).
  • Figure 2: L2 Capability vs. Susceptibility (why Utility/Safety matter). (Left) Neutral performance reflects baseline interventional judgment capability. (Right) Susceptibility measures label drift under nuisance pressure that should not flip the gold label. Social pressure is near-zero for most models on T3-L2, while epistemic pressure can trigger substantial reversals, revealing instability not captured by neutral accuracy alone.
  • Figure 3: L2 Dynamics of Self-Doubt. (Left) Bad flips versus good flips under interrogation, indicating whether "rethink" behaves like selective verification or indiscriminate reversal. (Right) Net impact on final accuracy, showing degradation when bad flips dominate.
  • Figure 4: The Scaling Paradox on L3 Ambiguity. (Top) Wolf Matrix (circles: Base; crosses: RCA-wrapped). The red dashed line highlights a 55 pp Safety gap between the older GPT-4-Turbo (75%) and the larger GPT-5.2 (20%) in the Base condition. (Bottom) RCA reduces the CONDITIONAL rate, effectively resolving the paralysis-under-uncertainty that drives the gap.