Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPs

TL;DR

This work generalizes the reward hypothesis by adopting a memoryless, lexicographic expected utility (EU) framework for sequential decision making. It identifies a simple condition–the presence of an infinitely bad unsafe outcome–that makes scalar rewards inadequate and induces a $2$-dimensional lexicographic reward, with a full $d$-dimensional extension via a recursive form $u(e\cdot\tau) = r(e) + \Gamma(e) u(\tau)$. The authors prove a fundamental theorem for Lexicographic MDPs (LMDPs), showing the existence of stationary, uniformly optimal policies and highlighting key differences from Constrained MDPs (CMDPs). They further develop both a single-unsafe-utility 2D characterization and its sequential extension, discuss AI-safety implications, and outline limitations and directions for future work. Overall, the framework generalizes scalar rewards while preserving core RL optimization principles in a prioritized, non-scalar setting.

Abstract

Recent work has formalized the reward hypothesis through the lens of expected utility theory, by interpreting reward as utility. Hausner's foundational work showed that dropping the continuity axiom leads to a generalization of expected utility theory where utilities are lexicographically ordered vectors of arbitrary dimension. In this paper, we extend this result by identifying a simple and practical condition under which preferences cannot be represented by scalar rewards, necessitating a 2-dimensional reward function. We provide a full characterization of such reward functions, as well as the general d-dimensional case, in Markov Decision Processes (MDPs) under a memorylessness assumption on preferences. Furthermore, we show that optimal policies in this setting retain many desirable properties of their scalar-reward counterparts, while in the Constrained MDP (CMDP) setting -- another common multiobjective setting -- they do not.

Figures (3)

• Figure 1: An example of lexicographic planning with time-horizon $T=4$ steps. The diagram depicts a $0.1$ probability of unsafety when transitioning from a green, red, or blue state. It also depicts the reward from gray states as $R$ and the optimal policy as black arrows. Starting from the green state, the optimal policy is to forgo the high reward on the right side in favor of increased safety and go left. Starting from the red state, since the safety of going left and right is the same, the optimal policy is to go right to get more cumulative reward. Importantly, decisions have to be based on two quantities: the probability of safety and expected cumulative reward.
• Figure 2: An example plotting the 2-dimensional lexicographic utility of $\mathcal{O} = \mleft\{o_1, ..., o_4, o^\dagger\mright\}$. Assuming $u$ is linear, its range will be the highlighted triangle. Note that \ref{['ax:continuity']} is the main axiom responsible for utilities being mapped onto a straight line. Since $o^\dagger$ does not satisfy \ref{['ax:continuity']}, it gets mapped outside the line containing the set of points $u(\mathcal{O}_\mathrm{safe})$.
• Figure 3: (Left) A set of policies $x$ (light cyan region), plotted by their $R, C$ values. The solutions of the constrained \ref{['c']}, penalty \ref{['p']}, and lexicographic \ref{['lex']} methods are indicated. The lowest-risk (safest) policy is given by \ref{['lex']}. (Right) The agent aims to reach the target. The cost $C$ is the path length, and the risk $R$ is the portion of the path that lies within the unsafe (red) region.

Theorems & Definitions (27)

• Definition 1
• Definition 2
• Theorem 1: vonneumann1947theory
• Definition 3
• Definition 4
• Theorem 2: hausner1954multidimensional
• Theorem 3: name=Sequential Lexicographic Expected Utility Theorem
• Theorem 4: name=Discounted Lexicographic Expected Utility Theorem
• Lemma 1
• Theorem 5: name=Lexicographic Expected Utility Theorem with a Single Unsafe Utility