Notes on the Reward Representation of Posterior Updates

TL;DR

This paper investigates KL-regularized optimization in control and inference under the boundary condition that the optimizer coincides with a Bayes posterior conditional under a single ambient joint distribution $P$. It derives a pointwise PMI-based representation of the identified interaction, showing that absolute rewards are only defined up to a context-dependent baseline (gauge) and that a commutativity constraint couples reward parametrizations across different conditioning orders. The main contributions are (i) the PMI-shaped representation $\alpha\bigl[r_z(x|y)+V(x,y,z)-V(y,z)\bigr]=i(x;z|y)$, (ii) the gauge freedom in decomposing rewards and values, and (iii) the order-robust coherence constraint that links different update directions. The results clarify the exact boundary where Bayesian conditioning directly informs policy updates, highlight the limits of reward identifiability from conditionals alone, and provide a principled blueprint for coherent design in information-constrained decision problems, with implications for psychology, economics, and control theory.

Abstract

Many ideas in modern control and reinforcement learning treat decision-making as inference: start from a baseline distribution and update it when a signal arrives. We ask when this can be made literal rather than metaphorical. We study the special case where a KL-regularized soft update is exactly a Bayesian posterior inside a single fixed probabilistic model, so the update variable is a genuine channel through which information is transmitted. In this regime, behavioral change is driven only by evidence carried by that channel: the update must be explainable as an evidence reweighing of the baseline. This yields a sharp identification result: posterior updates determine the relative, context-dependent incentive signal that shifts behavior, but they do not uniquely determine absolute rewards, which remain ambiguous up to context-specific baselines. Requiring one reusable continuation value across different update directions adds a further coherence constraint linking the reward descriptions associated with different conditioning orders.