Are LLMs Biased Like Humans? Causal Reasoning as a Function of Prior Knowledge, Irrelevant Information, and Reasoning Budget

TL;DR

The paper interrogates whether LLMs perform causal reasoning like normative models or rely on human-like shortcuts by evaluating 20+ LLMs against a human baseline on 11 collider-graph tasks. Using direct and chain-of-thought prompting, plus content-manipulation robustness tests, the authors show LLM judgments are largely describable by compact causalBayes nets with leaky-OR structure, and CoT prompting improves cross-task generalization and alignment with humans. Most LLMs exhibit stronger explaining away and are largely Markov-compliant, contrasting with human tendencies toward latent-factor attribution; robustness is highly model-dependent and often enhanced by CoT. The work demonstrates that Bayesian models provide valuable interpretive targets for diagnosing LLM reasoning and informs safe deployment, highlighting both complementary strengths and potential brittleness when uncertainty is intrinsic.

Abstract

Large language models (LLMs) are increasingly used in domains where causal reasoning matters, yet it remains unclear whether their judgments reflect normative causal computation, human-like shortcuts, or brittle pattern matching. We benchmark 20+ LLMs against a matched human baseline on 11 causal judgment tasks formalized by a collider structure ($C_1 \!\rightarrow\! E\! \leftarrow \!C_2$). We find that a small interpretable model compresses LLMs' causal judgments well and that most LLMs exhibit more rule-like reasoning strategies than humans who seem to account for unmentioned latent factors in their probability judgments. Furthermore, most LLMs do not mirror the characteristic human collider biases of weak explaining away and Markov violations. We probe LLMs' causal judgment robustness under (i) semantic abstraction and (ii) prompt overloading (injecting irrelevant text), and find that chain-of-thought (CoT) increases robustness for many LLMs. Together, this divergence suggests LLMs can complement humans when known biases are undesirable, but their rule-like reasoning may break down when uncertainty is intrinsic -- highlighting the need to characterize LLM reasoning strategies for safe, effective deployment.

Figures (9)

• Figure 1: \ref{['fig:sub_a']} and \ref{['fig:sub_b']} represent two out of 11 conditional probability (inference) queries. The colors of the nodes represent the three possible states the variables that the nodes of the graph can be in: $\!\to$ observed node $\!\in\! \{0,1\}$, $\!\to\!$ latent (query node), $\!\to\!$ no information. \ref{['fig:sub_c']} shows an example of one of the three domains (Sociology) used in the RW17 conditions that "verbalize" the graph, where each node is represented by a variable in the domain.
• Figure 2: Human--LLM alignment: CoT boosts alignment.
• Figure 3: Like humans, LLMs judge the effect as more likely in the presence of more causes (\ref{['fig:sub_pred_inference']}). Unlike humans, most LLMs (a) do not reproduce the Markov violation bias and instead respect the independence of causes (\ref{['fig:bias_mv']}) and (b) show strong explaining away where humans show only a weak effect (\ref{['fig:bias_ea']}).
• Figure 4: LLMs' probability judgments are well captured by an interpretable causal model, indicated by good causal Bayes net fits; CoT improves fit.
• Figure 5: Out-of-sample CBN generalization (LOOCV $R^2$); higher is better.