Incoherent Beliefs & Inconsistent Actions in Large Language Models

TL;DR

The paper investigates how large language models update beliefs and act on them in dynamic, interactive settings. By combining Bayesian updating tests, Metaculus betting experiments, and deference challenges across multiple models and confidence signals, it reveals substantial belief–action gaps and variable deference-consistency, even among strong models. It shows that belief updates often diverge from Bayes’ rule and that actions frequently do not align with beliefs, with only modest transfer of static-task performance or calibration to dynamic settings. The authors also demonstrate that prompting and activation steering can improve deference-consistency, underscoring that such behaviors are modifiable but not inherent. Overall, the work argues for evaluating LLMs in interactive, real-world-like environments and developing methods to mitigate belief–action misalignment.

Abstract

Real-world tasks and environments exhibit differences from the static datasets that large language models (LLMs) are typically evaluated on. Such tasks can involve sequential interaction, requiring coherent updating of beliefs in light of new evidence, and making appropriate decisions based on those beliefs. Predicting how LLMs will perform in such dynamic environments is important, but can be tricky to determine from measurements in static settings. In this work, we examine two critical components of LLM performance: the ability of LLMs to coherently update their beliefs, and the extent to which the actions they take are consistent with those beliefs. First, we find that LLMs are largely inconsistent in how they update their beliefs; models can exhibit up to a 30% average difference between the directly elicited posterior, and the correct update of their prior. Second, we find that LLMs also often take actions which are inconsistent with the beliefs they hold. On a betting market, for example, LLMs often do not even bet in the same direction as their internally held beliefs over the underlying outcomes. We also find they have moderate self-inconsistency in how they respond to challenges by users to given answers. Finally, we show that the above properties hold even for strong models that obtain high accuracy or that are well-calibrated on the tasks at hand. Our results highlight the difficulties of predicting LLM behavior in complex real-world settings.

Figures (18)

• Figure 1: An example of an LLM betting on the opposite side of its belief.
• Figure 2: An example of deference-inconsistency in models. Models may stick to answers that they have low confidence in, yet switch for answers with higher confidence.
• Figure 3: Brier scores $BS(p_2, p_2^*)$ describing the deviation between the model's directly elicited posterior $p_2$ and the Bayes-predicted posterior $p_2^*$ for a diabetes diagnosis given new evidence. The logit-derived confidences of all models deviate significantly from Bayes' theorem.
• Figure 4: Brier scores describing the correlation of each of $p_1$, $p_2$ and $p_2^*$ with the diagnosis $D$ as described in \ref{['sec:bayesian_updaters']}. For all models, except GPT-4o, the elicited posterior $p_2$ has worse predictive performance than the calculated posterior $p_2^*$, and even the elicited prior $p_1$.
• Figure 5: Mean distance from optimal betting for each model when prompted to maximize either linear or log utility, reported for logit and verbal confidence elicitation. Distances are plotted against expected distances for a no betting baseline (dark gray) and a 50% probability betting baseline. Most models perform worse than baseline.