Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making

Khurram Yamin, Jingjing Tang, Santiago Cortes-Gomez, Amit Sharma, Eric Horvitz, Bryan Wilder

TL;DR

This work develops a decision-theoretic framework to assess whether elicited probabilities from LLMs reflect subjective beliefs that drive actions under uncertainty. By deriving falsifiable conditions from random utility and prospect theory models, it links beliefs to choices via tests like $I(A;\theta\mid p)=0$ and monotone pairwise probabilities, and applies these to four medical-diagnosis tasks across multiple LLMs. Across real-world and expert-network data, the study finds systematic but imperfect alignment between elicited beliefs and decisions, with robust evidence of belief insufficiency and model-specific deviations. The results highlight how decision-oriented evaluation can diagnose and guide improvements in high-stakes LLM decision support, while clearly acknowledging that such tests falsify rather than prove rationality. Overall, the paper provides a practical, falsification-based toolkit for diagnosing decision-relevant beliefs in LLMs with implications for safety and reliability in medical contexts.

Abstract

Large language models (LLMs) are increasingly deployed as agents in high-stakes domains where optimal actions depend on both uncertainty about the world and consideration of utilities of different outcomes, yet their decision logic remains difficult to interpret. We study whether LLMs are rational utility maximizers with coherent beliefs and stable preferences. We consider behaviors of models for diagnosis challenge problems. The results provide insights about the relationship of LLM inferences to ideal Bayesian utility maximization for elicited probabilities and observed actions. Our approach provides falsifiable conditions under which the reported probabilities \emph{cannot} correspond to the true beliefs of any rational agent. We apply this methodology to multiple medical diagnostic domains with evaluations across several LLMs. We discuss implications of the results and directions forward for uses of LLMs in guiding high-stakes decisions.

Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making

TL;DR

This work develops a decision-theoretic framework to assess whether elicited probabilities from LLMs reflect subjective beliefs that drive actions under uncertainty. By deriving falsifiable conditions from random utility and prospect theory models, it links beliefs to choices via tests like and monotone pairwise probabilities, and applies these to four medical-diagnosis tasks across multiple LLMs. Across real-world and expert-network data, the study finds systematic but imperfect alignment between elicited beliefs and decisions, with robust evidence of belief insufficiency and model-specific deviations. The results highlight how decision-oriented evaluation can diagnose and guide improvements in high-stakes LLM decision support, while clearly acknowledging that such tests falsify rather than prove rationality. Overall, the paper provides a practical, falsification-based toolkit for diagnosing decision-relevant beliefs in LLMs with implications for safety and reliability in medical contexts.

Abstract

Large language models (LLMs) are increasingly deployed as agents in high-stakes domains where optimal actions depend on both uncertainty about the world and consideration of utilities of different outcomes, yet their decision logic remains difficult to interpret. We study whether LLMs are rational utility maximizers with coherent beliefs and stable preferences. We consider behaviors of models for diagnosis challenge problems. The results provide insights about the relationship of LLM inferences to ideal Bayesian utility maximization for elicited probabilities and observed actions. Our approach provides falsifiable conditions under which the reported probabilities \emph{cannot} correspond to the true beliefs of any rational agent. We apply this methodology to multiple medical diagnostic domains with evaluations across several LLMs. We discuss implications of the results and directions forward for uses of LLMs in guiding high-stakes decisions.
Paper Structure (54 sections, 2 theorems, 31 equations, 2 figures, 5 tables)

This paper contains 54 sections, 2 theorems, 31 equations, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Decision-Theoretic Framework
  5. Testable Implications for Belief Elicitation
  6. Conditional independence of actions and outcomes:
  7. Monotone pairwise choice probabilities.
  8. Experimental Setup
  9. Results
  10. Conditional Independence Test.
  11. Discussion and Conclusions
  12. Impact Statement
  13. Acknowledgements
  14. Proofs
  15. Proof of Proposition \ref{['thm:ci_rum']}
  16. ...and 39 more sections

Key Result

Proposition 3.3

For any agent satisfying Definition definition:rum or definition:pt, their subjective beliefs and actions satisfy $a \perp \theta \mid P_S(\theta\mid x)$. Conversely, if for a set of elicited beliefs $P_E$, $a \not\perp \theta \mid P_E(\theta|x)$, then there is no agent satisfying Definition definit

Figures (2)

  • Figure 1: Monotone Odds Violations: In these plots, we show examples of cases with and without statistically significant monotonicity violations. The segments without violations are in blue and those with are in red. The x-axis shows the elicited probability, and the y-axis shows choice probability ratio. We use Y to represent Yes $(a=1)$, N for No $(a=0)$, and D for Defer.
  • Figure 2: Internal Consistency Error via Law of Iterated Expectation (LIE): Here we plot the 95$\%$ CI for the median quantity $\Delta_{\text{LIE}}(x)$/$p(x)$ (see Eq \ref{['eq:lie_error']}) by dataset(subplots) and model(bars). A cross-validated random forest model that is separately trained on ground truth $p^*(x)$ and $p^*(x,z)$ (also conditional on next state distribution z) values is used as a baseline.

Theorems & Definitions (9)

  • Definition 3.1: Random Utility Model
  • Definition 3.2: PT-RUM
  • Proposition 3.3: Conditional Independence under Truthful Reporting
  • Remark 3.4
  • Definition 3.5: IIA
  • Proposition 3.6: Monotone pairwise choice probability
  • Remark 3.7
  • proof
  • proof