The Paradox of Robustness: Decoupling Rule-Based Logic from Affective Noise in High-Stakes Decision-Making

TL;DR

This study investigates whether instruction-tuned LLMs exhibit human-like framing biases in high-stakes, rule-bound decisions. Using a rigorously controlled perturbation framework and a 162-scenario benchmark across healthcare, finance, and law, the authors quantify narrative sensitivity through metrics like Decision Drift, Flip Rate, and output entropy, analyzed with BCa bootstrap and Bayes factors. Across six models, three affective-intensity levels, and three domains, LLMs show near-zero framing effects (aggregate $\Delta \approx -0.1\%$, 95% CI $[-2.3\%, +2.1\%]$, $\text{BF}_{01} \approx 109$), indicating a robust decoupling of logical rule-adherence from affective narratives. The work further demonstrates invariance to perturbation type, model training paradigm, and ablation of explicit ignore-narrative prompts, and extends robustness to multiclass decisions, suggesting a practical role for LLMs as stable institutional arbiters. The findings motivate mechanistic investigations into instruction-priority encodings and highlight methodological contributions (controlled perturbations, 162-scenario benchmark) that support reproducible evaluation of robustness in decision-critical AI systems.

Abstract

While Large Language Models (LLMs) are widely documented to be sensitive to minor prompt perturbations and prone to sycophantic alignment with user biases, their robustness in consequential, rule-bound decision-making remains under-explored. In this work, we uncover a striking "Paradox of Robustness": despite their known lexical brittleness, instruction-tuned LLMs exhibit a behavioral and near-total invariance to emotional framing effects. Using a novel controlled perturbation framework across three high-stakes domains (healthcare, law, and finance), we quantify a robustness gap where LLMs demonstrate 110-300 times greater resistance to narrative manipulation than human subjects. Specifically, we find a near-zero effect size for models (Cohen's h = 0.003) compared to the substantial biases observed in humans (Cohen's h in [0.3, 0.8]). This result is highly counterintuitive and suggests the mechanisms driving sycophancy and prompt sensitivity do not necessarily translate to a failure in logical constraint satisfaction. We show that this invariance persists across models with diverse training paradigms. Our findings show that while LLMs may be "brittle" to how a query is formatted, they are remarkably "stable" against why a decision should be biased. Our findings establish that instruction-tuned models can decouple logical rule-adherence from persuasive narratives, offering a source of decision stability that complements, and even potentially de-biases, human judgment in institutional contexts. We release the 162-scenario benchmark, code, and data to facilitate the rigorous evaluation of narrative-induced bias and robustness on GitHub.com.

Figures (10)

• Figure 1: Equivalence plot showing Decision Drift by model with 95% bootstrap CIs. The green Region of Practical Equivalence (ROPE, $\pm3\%$) defines negligible effects. All CIs fall within or overlap ROPE, demonstrating effects are practically zero.
• Figure 2: Decision Drift by affective intensity tier. Bars show point estimates with 95% bootstrap CIs. No monotonic dose-response relationship is observed; all intervals span zero (n.s.). Maximum emotional intensity ($\tau=4$) produces smaller drift than moderate ($\tau=2$).
• Figure 3: Forest plot of Decision Drift by scenario. All 9 scenario CIs span zero with $I^2 = 0.0\%$ heterogeneity. The diamond shows the pooled estimate. This consistency rules out scenario-specific vulnerabilities.
• Figure 4: Effect size comparison: Human framing effects from behavioral economics literature vs. LLM narrative sensitivity in this study. The LLM effect (Cohen's $h = 0.003$) is 110--300$\times$ smaller than typical human susceptibility.
• Figure 5: Decision Drift by training paradigm with 95% bootstrap CIs. All four paradigms---US RLHF (GPT-5-mini), Constitutional AI (Claude), Chinese Ecosystem (DeepSeek, Qwen), and Open-Source RLHF (Llama, Mistral)---show CIs spanning zero, indicating robustness is not paradigm-specific.